Sep 14 12:58:01.438026 ip-10-0-129-61 systemd[1]: Starting Kubernetes Kubelet...
Sep 14 12:58:01.879333 ip-10-0-129-61 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 12:58:01.879333 ip-10-0-129-61 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 14 12:58:01.879333 ip-10-0-129-61 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 12:58:01.879333 ip-10-0-129-61 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 14 12:58:01.879333 ip-10-0-129-61 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 12:58:01.882297 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.882201    2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 14 12:58:01.887372 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887356    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 12:58:01.887372 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887372    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 12:58:01.887434 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887377    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 12:58:01.887434 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887380    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 12:58:01.887434 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887383    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 12:58:01.887434 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887386    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 12:58:01.887434 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887389    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 12:58:01.887434 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887392    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 12:58:01.887434 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887394    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 12:58:01.887434 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887397    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 12:58:01.887434 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887399    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 12:58:01.887434 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887402    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 12:58:01.887434 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887404    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 12:58:01.887434 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887407    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 12:58:01.887434 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887410    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 12:58:01.887434 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887412    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 12:58:01.887434 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887415    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 12:58:01.887434 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887418    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 12:58:01.887434 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887420    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 12:58:01.887434 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887423    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 12:58:01.887434 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887426    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 12:58:01.887434 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887435    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 12:58:01.887917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887438    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 12:58:01.887917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887441    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 12:58:01.887917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887444    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 12:58:01.887917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887448    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 12:58:01.887917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887452    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 12:58:01.887917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887455    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 12:58:01.887917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887458    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 12:58:01.887917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887461    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 12:58:01.887917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887464    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 12:58:01.887917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887466    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 12:58:01.887917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887469    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 12:58:01.887917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887472    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 12:58:01.887917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887475    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 12:58:01.887917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887478    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 12:58:01.887917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887481    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 12:58:01.887917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887483    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 12:58:01.887917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887485    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 12:58:01.887917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887488    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 12:58:01.887917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887491    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 12:58:01.887917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887493    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 12:58:01.888406 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887496    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 12:58:01.888406 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887498    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 12:58:01.888406 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887501    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 12:58:01.888406 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887503    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 12:58:01.888406 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887506    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 12:58:01.888406 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887508    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 12:58:01.888406 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887510    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 12:58:01.888406 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887513    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 12:58:01.888406 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887516    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 12:58:01.888406 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887518    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 12:58:01.888406 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887520    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 12:58:01.888406 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887524    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 12:58:01.888406 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887528    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 12:58:01.888406 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887531    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 12:58:01.888406 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887534    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 12:58:01.888406 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887536    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 12:58:01.888406 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887539    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 12:58:01.888406 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887541    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 12:58:01.888406 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887544    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 12:58:01.888917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887546    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 12:58:01.888917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887549    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 12:58:01.888917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887551    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 12:58:01.888917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887555    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 12:58:01.888917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887557    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 12:58:01.888917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887560    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 12:58:01.888917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887562    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 12:58:01.888917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887565    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 12:58:01.888917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887567    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 12:58:01.888917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887570    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 12:58:01.888917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887572    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 12:58:01.888917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887575    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 12:58:01.888917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887577    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 12:58:01.888917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887580    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 12:58:01.888917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887582    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 12:58:01.888917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887585    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 12:58:01.888917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887588    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 12:58:01.888917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887590    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 12:58:01.888917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887593    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 12:58:01.888917 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887595    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 12:58:01.889478 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887598    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 12:58:01.889478 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887600    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 12:58:01.889478 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887602    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 12:58:01.889478 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887605    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 12:58:01.889478 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887607    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 12:58:01.889478 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887612    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 12:58:01.889478 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887617    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 12:58:01.889478 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887620    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 12:58:01.889478 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887623    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 12:58:01.889478 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887626    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 12:58:01.889478 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887629    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 12:58:01.889478 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887632    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 12:58:01.889478 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887634    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 12:58:01.889478 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887637    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 12:58:01.889478 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887640    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 12:58:01.889478 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887643    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 12:58:01.889478 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887645    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 12:58:01.889478 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.887648    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 12:58:01.889478 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889166    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 12:58:01.889960 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889173    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 12:58:01.889960 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889178    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 12:58:01.889960 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889182    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 12:58:01.889960 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889185    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 12:58:01.889960 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889188    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 12:58:01.889960 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889191    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 12:58:01.889960 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889194    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 12:58:01.889960 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889197    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 12:58:01.889960 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889200    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 12:58:01.889960 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889203    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 12:58:01.889960 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889205    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 12:58:01.889960 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889210    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 12:58:01.889960 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889212    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 12:58:01.889960 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889215    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 12:58:01.889960 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889218    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 12:58:01.889960 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889222    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 12:58:01.889960 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889226    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 12:58:01.889960 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889228    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 12:58:01.889960 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889231    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 12:58:01.889960 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889234    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 12:58:01.890453 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889238    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 12:58:01.890453 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889241    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 12:58:01.890453 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889244    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 12:58:01.890453 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889247    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 12:58:01.890453 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889250    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 12:58:01.890453 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889253    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 12:58:01.890453 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889256    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 12:58:01.890453 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889259    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 12:58:01.890453 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889261    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 12:58:01.890453 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889264    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 12:58:01.890453 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889266    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 12:58:01.890453 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889269    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 12:58:01.890453 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889271    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 12:58:01.890453 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889274    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 12:58:01.890453 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889277    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 12:58:01.890453 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889279    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 12:58:01.890453 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889282    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 12:58:01.890453 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889284    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 12:58:01.890453 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889287    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 12:58:01.890453 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889290    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 12:58:01.891156 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889293    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 12:58:01.891156 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889295    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 12:58:01.891156 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889298    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 12:58:01.891156 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889301    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 12:58:01.891156 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889303    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 12:58:01.891156 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889306    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 12:58:01.891156 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889308    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 12:58:01.891156 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889311    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 12:58:01.891156 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889314    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 12:58:01.891156 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889317    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 12:58:01.891156 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889319    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 12:58:01.891156 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889322    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 12:58:01.891156 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889325    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 12:58:01.891156 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889328    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 12:58:01.891156 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889331    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 12:58:01.891156 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889334    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 12:58:01.891156 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889336    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 12:58:01.891156 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889338    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 12:58:01.891156 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889341    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 12:58:01.891713 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889344    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 12:58:01.891713 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889346    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 12:58:01.891713 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889349    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 12:58:01.891713 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889351    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 12:58:01.891713 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889354    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 12:58:01.891713 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889357    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 12:58:01.891713 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889359    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 12:58:01.891713 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889362    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 12:58:01.891713 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889364    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 12:58:01.891713 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889367    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 12:58:01.891713 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889370    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 12:58:01.891713 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889373    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 12:58:01.891713 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889377    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 12:58:01.891713 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889379    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 12:58:01.891713 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889382    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 12:58:01.891713 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889385    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 12:58:01.891713 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889388    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 12:58:01.891713 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889390    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 12:58:01.891713 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889392    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 12:58:01.892205 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889395    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 12:58:01.892205 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889398    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 12:58:01.892205 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889400    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 12:58:01.892205 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889403    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 12:58:01.892205 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889405    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 12:58:01.892205 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889408    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 12:58:01.892205 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889411    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 12:58:01.892205 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889413    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 12:58:01.892205 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889416    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 12:58:01.892205 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889419    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 12:58:01.892205 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889422    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 12:58:01.892205 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889424    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 12:58:01.892205 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889427    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 12:58:01.892205 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889429    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 12:58:01.892205 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889432    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 12:58:01.892205 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889434    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 12:58:01.892205 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889437    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 12:58:01.892205 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889439    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 12:58:01.892205 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889442    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 12:58:01.892205 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889444    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 12:58:01.892692 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889800    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 12:58:01.892692 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889805    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 12:58:01.892692 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889808    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 12:58:01.892692 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889810    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 12:58:01.892692 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889813    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 12:58:01.892692 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889816    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 12:58:01.892692 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889818    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 12:58:01.892692 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889821    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 12:58:01.892692 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889824    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 12:58:01.892692 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889826    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 12:58:01.892692 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889829    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 12:58:01.892692 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889832    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 12:58:01.892692 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889835    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 12:58:01.892692 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889837    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 12:58:01.892692 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889840    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 12:58:01.892692 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889843    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 12:58:01.892692 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889846    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 12:58:01.892692 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889849    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 12:58:01.892692 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889852    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 12:58:01.892692 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889854    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 12:58:01.893217 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889857    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 12:58:01.893217 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889860    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 12:58:01.893217 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889863    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 12:58:01.893217 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889865    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 12:58:01.893217 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889868    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 12:58:01.893217 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889871    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 12:58:01.893217 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889873    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 12:58:01.893217 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889876    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 12:58:01.893217 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889878    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 12:58:01.893217 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889881    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 12:58:01.893217 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889883    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 12:58:01.893217 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889886    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 12:58:01.893217 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889889    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 12:58:01.893217 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889891    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 12:58:01.893217 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889894    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 12:58:01.893217 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889896    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 12:58:01.893217 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889899    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 12:58:01.893217 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889901    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 12:58:01.893217 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889904    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 12:58:01.893217 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889906    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 12:58:01.893698 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889910    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 12:58:01.893698 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889913    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 12:58:01.893698 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889915    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 12:58:01.893698 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889918    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 12:58:01.893698 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889920    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 12:58:01.893698 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889923    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 12:58:01.893698 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889925    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 12:58:01.893698 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889928    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 12:58:01.893698 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889931    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 12:58:01.893698 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889933    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 12:58:01.893698 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889937    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 12:58:01.893698 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889940    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 12:58:01.893698 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889944    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 12:58:01.893698 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889947    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 12:58:01.893698 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889949    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 12:58:01.893698 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889952    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 12:58:01.893698 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889955    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 12:58:01.893698 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889957    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 12:58:01.893698 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889960    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 12:58:01.893698 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889963    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 12:58:01.894230 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889965    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 12:58:01.894230 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889968    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 12:58:01.894230 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889970    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 12:58:01.894230 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889974    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 12:58:01.894230 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889977    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 12:58:01.894230 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889979    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 12:58:01.894230 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889982    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 12:58:01.894230 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889985    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 12:58:01.894230 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889987    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 12:58:01.894230 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889989    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 12:58:01.894230 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889992    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 12:58:01.894230 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889995    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 12:58:01.894230 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.889998    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 12:58:01.894230 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890000    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 12:58:01.894230 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890003    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 12:58:01.894230 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890008    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 12:58:01.894230 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890011    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 12:58:01.894230 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890014    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 12:58:01.894230 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890018    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 12:58:01.894699 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890021    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 12:58:01.894699 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890025    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 12:58:01.894699 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890029    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 12:58:01.894699 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890032    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 12:58:01.894699 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890035    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 12:58:01.894699 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890038    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 12:58:01.894699 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890041    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 12:58:01.894699 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890044    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 12:58:01.894699 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890047    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 12:58:01.894699 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890050    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 12:58:01.894699 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890053    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 12:58:01.894699 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890055    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 12:58:01.894699 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890058    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 12:58:01.894699 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890061    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 12:58:01.894699 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890063    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 12:58:01.894699 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890066    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 12:58:01.894699 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890069    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 12:58:01.894699 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890071    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 12:58:01.894699 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890075    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.890079    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890161    2557 flags.go:64] FLAG: --address="0.0.0.0"
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890202    2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890209    2557 flags.go:64] FLAG: --anonymous-auth="true"
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890214    2557 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890218    2557 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890222    2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890226    2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890229    2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890233    2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890236    2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890240    2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890243    2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890246    2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890249    2557 flags.go:64] FLAG: --cgroup-root=""
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890252    2557 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890254    2557 flags.go:64] FLAG: --client-ca-file=""
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890257    2557 flags.go:64] FLAG: --cloud-provider="external"
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890260    2557 flags.go:64] FLAG: --cluster-dns="[]"
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890264    2557 flags.go:64] FLAG: --cluster-domain=""
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890267    2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890270    2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890273    2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 14 12:58:01.895298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890277    2557 flags.go:64] FLAG: --container-log-max-files="5"
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890280    2557 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890283    2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890286    2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890289    2557 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890292    2557 flags.go:64] FLAG: --contention-profiling="false"
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890295    2557 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890298    2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890301    2557 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890303    2557 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890307    2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890310    2557 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890314    2557 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890317    2557 flags.go:64] FLAG: --enable-load-reader="false"
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890319    2557 flags.go:64] FLAG: --enable-server="true"
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890322    2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890327    2557 flags.go:64] FLAG: --event-burst="100"
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890330    2557 flags.go:64] FLAG: --event-qps="50"
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890333    2557 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890337    2557 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890339    2557 flags.go:64] FLAG: --eviction-hard=""
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890343    2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890346    2557 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890349    2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890352    2557 flags.go:64] FLAG: --eviction-soft=""
Sep 14 12:58:01.895901 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890354    2557 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890357    2557 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890360    2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890363    2557 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890366    2557 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890369    2557 flags.go:64] FLAG: --fail-swap-on="true"
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890372    2557 flags.go:64] FLAG: --feature-gates=""
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890376    2557 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890379    2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890385    2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890388    2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890391    2557 flags.go:64] FLAG: --healthz-port="10248"
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890394    2557 flags.go:64] FLAG: --help="false"
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890397    2557 flags.go:64] FLAG: --hostname-override="ip-10-0-129-61.ec2.internal"
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890400    2557 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890403    2557 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890406    2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890409    2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890412    2557 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890415    2557 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890418    2557 flags.go:64] FLAG: --image-service-endpoint=""
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890421    2557 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890423    2557 flags.go:64] FLAG: --kube-api-burst="100"
Sep 14 12:58:01.896490 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890426    2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890429    2557 flags.go:64] FLAG: --kube-api-qps="50"
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890432    2557 flags.go:64] FLAG: --kube-reserved=""
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890435    2557 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890438    2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890441    2557 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890443    2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890446    2557 flags.go:64] FLAG: --lock-file=""
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890449    2557 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890452    2557 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890455    2557 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890461    2557 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890464    2557 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890467    2557 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890470    2557 flags.go:64] FLAG: --logging-format="text"
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890472    2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890476    2557 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890479    2557 flags.go:64] FLAG: --manifest-url=""
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890483    2557 flags.go:64] FLAG: --manifest-url-header=""
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890488    2557 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890548    2557 flags.go:64] FLAG: --max-open-files="1000000"
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890558    2557 flags.go:64] FLAG: --max-pods="110"
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890563    2557 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890568    2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890633    2557 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 14 12:58:01.897084 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890638    2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890643    2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890648    2557 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890659    2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890670    2557 flags.go:64] FLAG: --node-status-max-images="50"
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890675    2557 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890679    2557 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890684    2557 flags.go:64] FLAG: --pod-cidr=""
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890689    2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890711    2557 flags.go:64] FLAG: --pod-manifest-path=""
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890716    2557 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890726    2557 flags.go:64] FLAG: --pods-per-core="0"
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890731    2557 flags.go:64] FLAG: --port="10250"
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890749    2557 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890755    2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-085e55c7cb36eee94"
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890760    2557 flags.go:64] FLAG: --qos-reserved=""
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890770    2557 flags.go:64] FLAG: --read-only-port="10255"
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890775    2557 flags.go:64] FLAG: --register-node="true"
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890780    2557 flags.go:64] FLAG: --register-with-taints=""
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890808    2557 flags.go:64] FLAG: --registry-burst="10"
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890813    2557 flags.go:64] FLAG: --registry-qps="5"
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890818    2557 flags.go:64] FLAG: --reserved-cpus=""
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890823    2557 flags.go:64] FLAG: --reserved-memory=""
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890828    2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890834    2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 14 12:58:01.897754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890841    2557 flags.go:64] FLAG: --rotate-certificates="false"
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890846    2557 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890856    2557 flags.go:64] FLAG: --runonce="false"
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890860    2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890866    2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890871    2557 flags.go:64] FLAG: --seccomp-default="false"
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890876    2557 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890880    2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890885    2557 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890890    2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890895    2557 flags.go:64] FLAG: --storage-driver-password="root"
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890904    2557 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890909    2557 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890914    2557 flags.go:64] FLAG: --storage-driver-user="root"
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890919    2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890924    2557 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890929    2557 flags.go:64] FLAG: --system-cgroups=""
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890935    2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890949    2557 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890954    2557 flags.go:64] FLAG: --tls-cert-file=""
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890959    2557 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890966    2557 flags.go:64] FLAG: --tls-min-version=""
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890971    2557 flags.go:64] FLAG: --tls-private-key-file=""
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890976    2557 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890981    2557 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 14 12:58:01.898356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890985    2557 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 14 12:58:01.899002 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.890995    2557 flags.go:64] FLAG: --v="2"
Sep 14 12:58:01.899002 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.891001    2557 flags.go:64] FLAG: --version="false"
Sep 14 12:58:01.899002 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.891010    2557 flags.go:64] FLAG: --vmodule=""
Sep 14 12:58:01.899002 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.891016    2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 14 12:58:01.899002 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.891021    2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 14 12:58:01.899002 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891363    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 12:58:01.899002 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891373    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 12:58:01.899002 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891378    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 12:58:01.899002 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891383    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 12:58:01.899002 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891387    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 12:58:01.899002 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891391    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 12:58:01.899002 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891395    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 12:58:01.899002 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891399    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 12:58:01.899002 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891408    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 12:58:01.899002 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891412    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 12:58:01.899002 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891417    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 12:58:01.899002 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891421    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 12:58:01.899002 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891480    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 12:58:01.899002 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891604    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 12:58:01.899002 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891610    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 12:58:01.899002 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891615    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 12:58:01.899520 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891618    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 12:58:01.899520 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891645    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 12:58:01.899520 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891649    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 12:58:01.899520 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891653    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 12:58:01.899520 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891656    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 12:58:01.899520 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891659    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 12:58:01.899520 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891662    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 12:58:01.899520 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891665    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 12:58:01.899520 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891668    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 12:58:01.899520 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891671    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 12:58:01.899520 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891675    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 12:58:01.899520 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891678    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 12:58:01.899520 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891680    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 12:58:01.899520 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891683    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 12:58:01.899520 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891686    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 12:58:01.899520 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891689    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 12:58:01.899520 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891692    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 12:58:01.899520 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891694    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 12:58:01.899520 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891697    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 12:58:01.899520 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891700    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 12:58:01.900025 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891702    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 12:58:01.900025 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891705    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 12:58:01.900025 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891709    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 12:58:01.900025 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891711    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 12:58:01.900025 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891714    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 12:58:01.900025 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891717    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 12:58:01.900025 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891720    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 12:58:01.900025 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891724    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 12:58:01.900025 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891727    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 12:58:01.900025 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891731    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 12:58:01.900025 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891734    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 12:58:01.900025 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891756    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 12:58:01.900025 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891761    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 12:58:01.900025 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891768    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 12:58:01.900025 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891772    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 12:58:01.900025 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891775    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 12:58:01.900025 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891778    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 12:58:01.900025 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891781    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 12:58:01.900025 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891783    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 12:58:01.900025 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891786    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 12:58:01.900508 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891789    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 12:58:01.900508 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891792    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 12:58:01.900508 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891795    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 12:58:01.900508 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891797    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 12:58:01.900508 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891800    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 12:58:01.900508 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891802    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 12:58:01.900508 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891805    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 12:58:01.900508 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891808    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 12:58:01.900508 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891811    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 12:58:01.900508 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891814    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 12:58:01.900508 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891816    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 12:58:01.900508 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891819    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 12:58:01.900508 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891821    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 12:58:01.900508 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891824    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 12:58:01.900508 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891826    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 12:58:01.900508 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891829    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 12:58:01.900508 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891832    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 12:58:01.900508 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891834    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 12:58:01.900508 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891837    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 12:58:01.900508 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891839    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 12:58:01.901009 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891842    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 12:58:01.901009 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891846    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 12:58:01.901009 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891848    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 12:58:01.901009 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891851    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 12:58:01.901009 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891854    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 12:58:01.901009 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891856    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 12:58:01.901009 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891859    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 12:58:01.901009 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891863    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 12:58:01.901009 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891866    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 12:58:01.901009 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891869    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 12:58:01.901009 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891872    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 12:58:01.901009 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891875    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 12:58:01.901009 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891879    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 12:58:01.901009 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891882    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 12:58:01.901009 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891884    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 12:58:01.901009 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891887    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 12:58:01.901009 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891889    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 12:58:01.901009 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891892    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 12:58:01.901009 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891895    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 12:58:01.901009 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891897    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 12:58:01.901528 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891900    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 12:58:01.901528 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891904    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 12:58:01.901528 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.891908    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 12:58:01.901528 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.892716    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 12:58:01.901528 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.900315    2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 14 12:58:01.901528 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.900329    2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 14 12:58:01.901528 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.901180    2557 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 14 12:58:01.901528 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.901188    2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 14 12:58:01.901528 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901241    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 12:58:01.901528 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901246    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 12:58:01.901528 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901250    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 12:58:01.901528 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901254    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 12:58:01.901528 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901257    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 12:58:01.901528 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901260    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 12:58:01.901901 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901263    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 12:58:01.901901 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901266    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 12:58:01.901901 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901269    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 12:58:01.901901 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901271    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 12:58:01.901901 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901274    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 12:58:01.901901 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901277    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 12:58:01.901901 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901279    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 12:58:01.901901 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901282    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 12:58:01.901901 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901285    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 12:58:01.901901 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901287    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 12:58:01.901901 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901290    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 12:58:01.901901 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901293    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 12:58:01.901901 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901296    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 12:58:01.901901 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901298    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 12:58:01.901901 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901301    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 12:58:01.901901 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901304    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 12:58:01.901901 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901307    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 12:58:01.901901 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901309    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 12:58:01.901901 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901312    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 12:58:01.901901 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901315    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 12:58:01.902382 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901318    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 12:58:01.902382 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901320    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 12:58:01.902382 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901323    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 12:58:01.902382 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901328    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 12:58:01.902382 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901331    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 12:58:01.902382 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901334    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 12:58:01.902382 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901336    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 12:58:01.902382 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901339    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 12:58:01.902382 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901342    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 12:58:01.902382 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901344    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 12:58:01.902382 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901347    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 12:58:01.902382 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901349    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 12:58:01.902382 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901352    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 12:58:01.902382 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901355    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 12:58:01.902382 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901357    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 12:58:01.902382 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901360    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 12:58:01.902382 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901362    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 12:58:01.902382 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901365    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 12:58:01.902382 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901369    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 12:58:01.902865 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901372    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 12:58:01.902865 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901375    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 12:58:01.902865 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901378    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 12:58:01.902865 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901380    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 12:58:01.902865 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901383    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 12:58:01.902865 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901385    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 12:58:01.902865 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901388    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 12:58:01.902865 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901391    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 12:58:01.902865 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901395    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 12:58:01.902865 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901398    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 12:58:01.902865 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901401    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 12:58:01.902865 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901404    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 12:58:01.902865 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901407    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 12:58:01.902865 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901410    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 12:58:01.902865 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901412    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 12:58:01.902865 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901415    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 12:58:01.902865 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901418    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 12:58:01.902865 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901421    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 12:58:01.902865 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901424    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 12:58:01.902865 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901427    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 12:58:01.903359 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901430    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 12:58:01.903359 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901432    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 12:58:01.903359 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901434    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 12:58:01.903359 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901437    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 12:58:01.903359 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901440    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 12:58:01.903359 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901443    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 12:58:01.903359 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901446    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 12:58:01.903359 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901448    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 12:58:01.903359 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901451    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 12:58:01.903359 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901453    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 12:58:01.903359 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901456    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 12:58:01.903359 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901458    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 12:58:01.903359 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901461    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 12:58:01.903359 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901463    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 12:58:01.903359 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901466    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 12:58:01.903359 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901468    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 12:58:01.903359 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901471    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 12:58:01.903359 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901474    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 12:58:01.903359 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901477    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 12:58:01.903830 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901479    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 12:58:01.903830 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901482    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 12:58:01.903830 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901485    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 12:58:01.903830 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901487    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 12:58:01.903830 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901490    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 12:58:01.903830 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901492    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 12:58:01.903830 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901495    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 12:58:01.903830 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901497    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 12:58:01.903830 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901500    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 12:58:01.903830 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901503    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 12:58:01.903830 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901506    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 12:58:01.903830 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901510    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 12:58:01.903830 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901514    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 12:58:01.903830 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901517    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 12:58:01.903830 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901520    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 12:58:01.904193 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.901525    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 12:58:01.904193 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901631    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 12:58:01.904193 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901636    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 12:58:01.904193 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901639    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 12:58:01.904193 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901642    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 12:58:01.904193 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901646    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 12:58:01.904193 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901648    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 12:58:01.904193 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901652    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 12:58:01.904193 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901654    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 12:58:01.904193 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901657    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 12:58:01.904193 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901660    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 12:58:01.904193 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901662    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 12:58:01.904193 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901665    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 12:58:01.904193 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901667    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 12:58:01.904193 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901670    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 12:58:01.904556 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901673    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 12:58:01.904556 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901675    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 12:58:01.904556 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901678    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 12:58:01.904556 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901681    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 12:58:01.904556 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901683    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 12:58:01.904556 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901686    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 12:58:01.904556 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901688    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 12:58:01.904556 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901691    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 12:58:01.904556 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901693    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 12:58:01.904556 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901696    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 12:58:01.904556 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901699    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 12:58:01.904556 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901701    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 12:58:01.904556 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901704    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 12:58:01.904556 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901707    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 12:58:01.904556 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901710    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 12:58:01.904556 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901713    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 12:58:01.904556 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901715    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 12:58:01.904556 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901718    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 12:58:01.904556 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901721    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 12:58:01.904556 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901723    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 12:58:01.905067 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901726    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 12:58:01.905067 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901728    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 12:58:01.905067 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901731    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 12:58:01.905067 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901733    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 12:58:01.905067 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901750    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 12:58:01.905067 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901754    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 12:58:01.905067 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901756    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 12:58:01.905067 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901759    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 12:58:01.905067 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901762    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 12:58:01.905067 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901764    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 12:58:01.905067 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901767    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 12:58:01.905067 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901770    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 12:58:01.905067 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901772    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 12:58:01.905067 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901775    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 12:58:01.905067 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901778    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 12:58:01.905067 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901781    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 12:58:01.905067 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901783    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 12:58:01.905067 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901786    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 12:58:01.905067 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901789    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 12:58:01.905067 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901791    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 12:58:01.905558 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901794    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 12:58:01.905558 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901796    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 12:58:01.905558 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901799    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 12:58:01.905558 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901803    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 12:58:01.905558 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901806    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 12:58:01.905558 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901809    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 12:58:01.905558 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901813    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 12:58:01.905558 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901820    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 12:58:01.905558 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901824    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 12:58:01.905558 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901826    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 12:58:01.905558 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901829    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 12:58:01.905558 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901832    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 12:58:01.905558 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901834    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 12:58:01.905558 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901837    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 12:58:01.905558 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901839    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 12:58:01.905558 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901842    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 12:58:01.905558 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901845    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 12:58:01.905558 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901847    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 12:58:01.905558 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901850    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 12:58:01.905558 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901852    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 12:58:01.906059 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901855    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 12:58:01.906059 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901858    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 12:58:01.906059 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901860    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 12:58:01.906059 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901863    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 12:58:01.906059 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901865    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 12:58:01.906059 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901868    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 12:58:01.906059 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901872    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 12:58:01.906059 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901875    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 12:58:01.906059 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901879    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 12:58:01.906059 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901882    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 12:58:01.906059 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901886    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 12:58:01.906059 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901888    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 12:58:01.906059 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901892    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 12:58:01.906059 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901896    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 12:58:01.906059 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901899    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 12:58:01.906059 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901902    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 12:58:01.906059 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901905    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 12:58:01.906059 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901908    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 12:58:01.906059 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901911    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 12:58:01.906554 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901913    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 12:58:01.906554 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901917    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 12:58:01.906554 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901920    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 12:58:01.906554 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901923    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 12:58:01.906554 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901925    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 12:58:01.906554 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:01.901928    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 12:58:01.906554 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.901932    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 12:58:01.906554 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.902041    2557 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 14 12:58:01.906554 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.905222    2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 14 12:58:01.906554 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.906298    2557 server.go:1019] "Starting client certificate rotation"
Sep 14 12:58:01.906554 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.906391    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 14 12:58:01.906554 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.906419    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 14 12:58:01.909149 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.909136    2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 14 12:58:01.913061 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.913045    2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 14 12:58:01.925529 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.925509    2557 log.go:25] "Validated CRI v1 runtime API"
Sep 14 12:58:01.928826 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.928813    2557 log.go:25] "Validated CRI v1 image API"
Sep 14 12:58:01.929610 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.929593    2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 14 12:58:01.935673 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.935637    2557 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b9bb2525-eeac-4072-8cad-d8e434f97871:/dev/nvme0n1p3 f6ded320-05ec-477f-90cb-cefe6a674909:/dev/nvme0n1p4]
Sep 14 12:58:01.935781 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.935668    2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 14 12:58:01.936487 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.936470    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 14 12:58:01.941107 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.940908    2557 manager.go:216] Machine: {Timestamp:2026-09-14 12:58:01.939549378 +0000 UTC m=+0.374224888 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099095 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2257d0b0da74bcdc9706b2c6ee572e SystemUUID:ec2257d0-b0da-74bc-dc97-06b2c6ee572e BootID:025c3b19-7867-47f2-bc3f-acfe71ab4d3c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d3:e3:14:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d3:e3:14:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:92:ab:28:43:cc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 14 12:58:01.941107 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.941098    2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 14 12:58:01.941248 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.941177    2557 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 14 12:58:01.944631 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.944607    2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 14 12:58:01.944803 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.944636    2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-61.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 14 12:58:01.945547 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.945535    2557 topology_manager.go:138] "Creating topology manager with none policy"
Sep 14 12:58:01.945608 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.945553    2557 container_manager_linux.go:309] "Creating device plugin manager"
Sep 14 12:58:01.945608 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.945565    2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 14 12:58:01.945608 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.945585    2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 14 12:58:01.946879 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.946865    2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 14 12:58:01.951636 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.951622    2557 state_mem.go:36] "Initialized new in-memory state store"
Sep 14 12:58:01.951779 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.951768    2557 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 14 12:58:01.951838 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.951805    2557 kubelet.go:486] "Attempting to sync node with API server"
Sep 14 12:58:01.951838 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.951835    2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 14 12:58:01.951933 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.951854    2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 14 12:58:01.951933 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.951869    2557 kubelet.go:398] "Adding apiserver pod source"
Sep 14 12:58:01.951933 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.951882    2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 14 12:58:01.952225 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.952211    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 14 12:58:01.952285 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.952235    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 14 12:58:01.955891 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.955871    2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 14 12:58:01.958269 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.958256    2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 14 12:58:01.958308 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.958286    2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 14 12:58:01.960356 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.960345    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 14 12:58:01.960388 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.960361    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 14 12:58:01.960388 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.960368    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 14 12:58:01.960388 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.960373    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 14 12:58:01.960388 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.960382    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 14 12:58:01.960388 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.960387    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 14 12:58:01.960522 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.960406    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 14 12:58:01.960522 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.960412    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 14 12:58:01.960522 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.960419    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 14 12:58:01.960522 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.960424    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 14 12:58:01.960522 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.960432    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 14 12:58:01.960522 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.960437    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 14 12:58:01.961401 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.961392    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 14 12:58:01.961401 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.961401    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 14 12:58:01.963204 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:01.963179    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 14 12:58:01.963243 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:01.963185    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-61.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 14 12:58:01.963363 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.963345    2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-61.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 14 12:58:01.965023 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.965011    2557 server.go:1268] "Started kubelet"
Sep 14 12:58:01.965105 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.965056    2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 14 12:58:01.965170 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.965088    2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 14 12:58:01.965170 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.965146    2557 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 14 12:58:01.965731 ip-10-0-129-61 systemd[1]: Started Kubernetes Kubelet.
Sep 14 12:58:01.966717 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.966699    2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 14 12:58:01.967042 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.967031    2557 server.go:310] "Adding debug handlers to kubelet server"
Sep 14 12:58:01.970213 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:01.969870    2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 14 12:58:01.974772 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.974750    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 14 12:58:01.975380 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.975365    2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 14 12:58:01.975584 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:01.975545    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-61.ec2.internal\" not found"
Sep 14 12:58:01.976159 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.976136    2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 14 12:58:01.976228 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.976184    2557 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 14 12:58:01.976228 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.976196    2557 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 14 12:58:01.976298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.976272    2557 reconstruct.go:97] "Volume reconstruction finished"
Sep 14 12:58:01.976298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.976280    2557 reconciler.go:29] "Reconciler: start to sync state"
Sep 14 12:58:01.976982 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.976966    2557 factory.go:153] Registering CRI-O factory
Sep 14 12:58:01.976982 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.976982    2557 factory.go:223] Registration of the crio container factory successfully
Sep 14 12:58:01.977081 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.977026    2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 14 12:58:01.977081 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.977032    2557 factory.go:55] Registering systemd factory
Sep 14 12:58:01.977081 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.977038    2557 factory.go:223] Registration of the systemd container factory successfully
Sep 14 12:58:01.977081 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.977049    2557 factory.go:103] Registering Raw factory
Sep 14 12:58:01.977081 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.977058    2557 manager.go:1196] Started watching for new ooms in manager
Sep 14 12:58:01.977409 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:01.976191    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-61.ec2.internal.18d531583aeaada8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-61.ec2.internal,UID:ip-10-0-129-61.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-61.ec2.internal,},FirstTimestamp:2026-09-14 12:58:01.964981672 +0000 UTC m=+0.399657182,LastTimestamp:2026-09-14 12:58:01.964981672 +0000 UTC m=+0.399657182,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-61.ec2.internal,}"
Sep 14 12:58:01.977973 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.977940    2557 manager.go:349] Starting recovery of all containers
Sep 14 12:58:01.988921 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.988894    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lcv29"
Sep 14 12:58:01.989044 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:01.988972    2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-61.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 14 12:58:01.989831 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:01.989806    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 14 12:58:01.990903 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.990887    2557 manager.go:354] Recovery completed
Sep 14 12:58:01.994779 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.994765    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lcv29"
Sep 14 12:58:01.996757 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.996725    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 12:58:01.999340 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.999324    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 12:58:01.999412 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.999347    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 12:58:01.999412 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.999357    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-61.ec2.internal" event="NodeHasSufficientPID"
Sep 14 12:58:01.999965 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.999950    2557 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 14 12:58:01.999965 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.999965    2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 14 12:58:02.000081 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.999951    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 12:58:02.000081 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:01.999982    2557 state_mem.go:36] "Initialized new in-memory state store"
Sep 14 12:58:02.000940 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.000927    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 12:58:02.000996 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.000953    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 12:58:02.000996 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.000963    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-61.ec2.internal" event="NodeHasSufficientPID"
Sep 14 12:58:02.002104 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.002087    2557 policy_none.go:49] "None policy: Start"
Sep 14 12:58:02.002153 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.002111    2557 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 14 12:58:02.002153 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.002124    2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 14 12:58:02.004228 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.004215    2557 policy_none.go:47] "Start"
Sep 14 12:58:02.035311 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.035289    2557 manager.go:337] "Starting Device Plugin manager"
Sep 14 12:58:02.053411 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:02.035324    2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 14 12:58:02.053411 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.035337    2557 server.go:87] "Starting device plugin registration server"
Sep 14 12:58:02.053411 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.035624    2557 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 14 12:58:02.053411 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.035638    2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 14 12:58:02.053411 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.035992    2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 14 12:58:02.053411 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.036115    2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 14 12:58:02.053411 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.036124    2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 14 12:58:02.053411 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:02.036271    2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 14 12:58:02.053411 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:02.036314    2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-61.ec2.internal\" not found"
Sep 14 12:58:02.053411 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.040789    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 14 12:58:02.053411 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.041950    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 14 12:58:02.053411 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.041997    2557 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 14 12:58:02.053411 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.042018    2557 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 14 12:58:02.053411 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:02.042058    2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 14 12:58:02.053411 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.044338    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 12:58:02.136690 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.136625    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 12:58:02.137369 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.137354    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 12:58:02.137424 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.137378    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 12:58:02.137424 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.137388    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-61.ec2.internal" event="NodeHasSufficientPID"
Sep 14 12:58:02.137424 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.137405    2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-61.ec2.internal"
Sep 14 12:58:02.142501 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.142486    2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-61.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-61.ec2.internal"]
Sep 14 12:58:02.142558 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.142540    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 12:58:02.143942 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.143928    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 12:58:02.143994 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.143952    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 12:58:02.143994 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.143961    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-61.ec2.internal" event="NodeHasSufficientPID"
Sep 14 12:58:02.145243 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.145221    2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-61.ec2.internal"
Sep 14 12:58:02.145243 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:02.145242    2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-61.ec2.internal\": node \"ip-10-0-129-61.ec2.internal\" not found"
Sep 14 12:58:02.146295 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.146280    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 12:58:02.146380 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.146354    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-61.ec2.internal"
Sep 14 12:58:02.146424 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.146383    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 12:58:02.147684 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.147669    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 12:58:02.147780 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.147695    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 12:58:02.147780 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.147706    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-61.ec2.internal" event="NodeHasSufficientPID"
Sep 14 12:58:02.147780 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.147674    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 12:58:02.147780 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.147775    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 12:58:02.147922 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.147786    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-61.ec2.internal" event="NodeHasSufficientPID"
Sep 14 12:58:02.150358 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.150341    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-61.ec2.internal"
Sep 14 12:58:02.150436 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.150365    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 12:58:02.151104 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.151086    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 12:58:02.151175 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.151117    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 12:58:02.151175 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.151131    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-61.ec2.internal" event="NodeHasSufficientPID"
Sep 14 12:58:02.168774 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:02.168754    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-61.ec2.internal\" not found"
Sep 14 12:58:02.171255 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:02.171230    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-61.ec2.internal\" not found" node="ip-10-0-129-61.ec2.internal"
Sep 14 12:58:02.177020 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:02.177002    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-61.ec2.internal\" not found" node="ip-10-0-129-61.ec2.internal"
Sep 14 12:58:02.269692 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:02.269671    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-61.ec2.internal\" not found"
Sep 14 12:58:02.276990 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.276971    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0633c83db9ecaf6a98b80b3d362c853b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-61.ec2.internal\" (UID: \"0633c83db9ecaf6a98b80b3d362c853b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-61.ec2.internal"
Sep 14 12:58:02.277064 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.276994    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0633c83db9ecaf6a98b80b3d362c853b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-61.ec2.internal\" (UID: \"0633c83db9ecaf6a98b80b3d362c853b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-61.ec2.internal"
Sep 14 12:58:02.277064 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.277009    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f85c20039e12da5048d2ed5c7f69bdf5-config\") pod \"kube-apiserver-proxy-ip-10-0-129-61.ec2.internal\" (UID: \"f85c20039e12da5048d2ed5c7f69bdf5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-61.ec2.internal"
Sep 14 12:58:02.370181 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:02.370154    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-61.ec2.internal\" not found"
Sep 14 12:58:02.377384 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.377369    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f85c20039e12da5048d2ed5c7f69bdf5-config\") pod \"kube-apiserver-proxy-ip-10-0-129-61.ec2.internal\" (UID: \"f85c20039e12da5048d2ed5c7f69bdf5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-61.ec2.internal"
Sep 14 12:58:02.377450 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.377438    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0633c83db9ecaf6a98b80b3d362c853b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-61.ec2.internal\" (UID: \"0633c83db9ecaf6a98b80b3d362c853b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-61.ec2.internal"
Sep 14 12:58:02.377502 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.377467    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0633c83db9ecaf6a98b80b3d362c853b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-61.ec2.internal\" (UID: \"0633c83db9ecaf6a98b80b3d362c853b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-61.ec2.internal"
Sep 14 12:58:02.377560 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.377508    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0633c83db9ecaf6a98b80b3d362c853b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-61.ec2.internal\" (UID: \"0633c83db9ecaf6a98b80b3d362c853b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-61.ec2.internal"
Sep 14 12:58:02.377560 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.377475    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f85c20039e12da5048d2ed5c7f69bdf5-config\") pod \"kube-apiserver-proxy-ip-10-0-129-61.ec2.internal\" (UID: \"f85c20039e12da5048d2ed5c7f69bdf5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-61.ec2.internal"
Sep 14 12:58:02.377560 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.377553    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0633c83db9ecaf6a98b80b3d362c853b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-61.ec2.internal\" (UID: \"0633c83db9ecaf6a98b80b3d362c853b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-61.ec2.internal"
Sep 14 12:58:02.470841 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:02.470783    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-61.ec2.internal\" not found"
Sep 14 12:58:02.475941 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.475929    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-61.ec2.internal"
Sep 14 12:58:02.481569 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.481179    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-61.ec2.internal"
Sep 14 12:58:02.570888 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:02.570866    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-61.ec2.internal\" not found"
Sep 14 12:58:02.671383 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:02.671360    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-61.ec2.internal\" not found"
Sep 14 12:58:02.771844 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:02.771811    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-61.ec2.internal\" not found"
Sep 14 12:58:02.840649 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.840623    2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 12:58:02.876361 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.876324    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-61.ec2.internal"
Sep 14 12:58:02.890718 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.890694    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 14 12:58:02.892794 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.892779    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-61.ec2.internal"
Sep 14 12:58:02.901564 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.901546    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 14 12:58:02.905855 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.905838    2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 14 12:58:02.905971 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.905950    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 14 12:58:02.906030 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.905994    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 14 12:58:02.952161 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.952140    2557 apiserver.go:52] "Watching apiserver"
Sep 14 12:58:02.959780 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.959761    2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 12:58:02.969260 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.969238    2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 14 12:58:02.970428 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.970408    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-4xdjj","kube-system/global-pull-secret-syncer-zgj9g","kube-system/kube-apiserver-proxy-ip-10-0-129-61.ec2.internal","openshift-image-registry/node-ca-5vcr5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-61.ec2.internal","openshift-multus/multus-cz95j","openshift-multus/network-metrics-daemon-nl2vh","openshift-network-diagnostics/network-check-target-jcjld","openshift-ovn-kubernetes/ovnkube-node-jwqnf","kube-system/konnectivity-agent-b97dp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv","openshift-cluster-node-tuning-operator/tuned-xkznw","openshift-dns/node-resolver-nhvd2","openshift-multus/multus-additional-cni-plugins-zcv75"]
Sep 14 12:58:02.972960 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.972943    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:02.973047 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:02.973025    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nl2vh" podUID="898a4541-4e3f-4276-b037-ff179ac5fc52"
Sep 14 12:58:02.975281 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.975262    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 14 12:58:02.975422 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.975403    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:02.977637 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.977618    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:02.978451 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.978437    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 14 12:58:02.979422 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.979402    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 14 12:58:02.979422 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.979415    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 14 12:58:02.979555 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.979440    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 14 12:58:02.979555 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.979489    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 14 12:58:02.979555 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.979498    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-57lj5\""
Sep 14 12:58:02.979706 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.979579    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 14 12:58:02.979822 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.979796    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6hfq\" (UniqueName: \"kubernetes.io/projected/898a4541-4e3f-4276-b037-ff179ac5fc52-kube-api-access-t6hfq\") pod \"network-metrics-daemon-nl2vh\" (UID: \"898a4541-4e3f-4276-b037-ff179ac5fc52\") " pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:02.979910 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.979836    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-host-slash\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:02.979910 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.979859    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-node-log\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:02.979910 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.979885    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:02.980021 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.979913    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-run-systemd\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:02.980021 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.979929    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-etc-openvswitch\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:02.980021 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.979943    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-log-socket\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:02.980021 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.979978    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62d953d6-3b86-4c95-9a47-9579d88993a7-env-overrides\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:02.980180 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.980027    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f26cce38-f34e-4f48-9a02-88e6674fe266-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zcv75\" (UID: \"f26cce38-f34e-4f48-9a02-88e6674fe266\") " pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:02.980180 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.980054    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-host-kubelet\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:02.980180 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.980077    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-run-openvswitch\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:02.980180 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.980095    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-host-run-ovn-kubernetes\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:02.980180 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.980118    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g564q\" (UniqueName: \"kubernetes.io/projected/62d953d6-3b86-4c95-9a47-9579d88993a7-kube-api-access-g564q\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:02.980180 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.980139    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f26cce38-f34e-4f48-9a02-88e6674fe266-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zcv75\" (UID: \"f26cce38-f34e-4f48-9a02-88e6674fe266\") " pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:02.980180 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.980161    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhlrp\" (UniqueName: \"kubernetes.io/projected/f26cce38-f34e-4f48-9a02-88e6674fe266-kube-api-access-vhlrp\") pod \"multus-additional-cni-plugins-zcv75\" (UID: \"f26cce38-f34e-4f48-9a02-88e6674fe266\") " pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:02.980491 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.980214    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-host-cni-netd\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:02.980491 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.980248    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f26cce38-f34e-4f48-9a02-88e6674fe266-cnibin\") pod \"multus-additional-cni-plugins-zcv75\" (UID: \"f26cce38-f34e-4f48-9a02-88e6674fe266\") " pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:02.980491 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.980276    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-systemd-units\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:02.980491 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.980300    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62d953d6-3b86-4c95-9a47-9579d88993a7-ovnkube-config\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:02.980491 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.980321    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62d953d6-3b86-4c95-9a47-9579d88993a7-ovnkube-script-lib\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:02.980491 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.980344    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f26cce38-f34e-4f48-9a02-88e6674fe266-cni-binary-copy\") pod \"multus-additional-cni-plugins-zcv75\" (UID: \"f26cce38-f34e-4f48-9a02-88e6674fe266\") " pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:02.980491 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.980360    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs\") pod \"network-metrics-daemon-nl2vh\" (UID: \"898a4541-4e3f-4276-b037-ff179ac5fc52\") " pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:02.980491 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.980395    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-run-ovn\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:02.980491 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.980416    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-host-cni-bin\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:02.980491 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.980432    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62d953d6-3b86-4c95-9a47-9579d88993a7-ovn-node-metrics-cert\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:02.980491 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.980449    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f26cce38-f34e-4f48-9a02-88e6674fe266-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zcv75\" (UID: \"f26cce38-f34e-4f48-9a02-88e6674fe266\") " pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:02.980491 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.980463    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-host-run-netns\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:02.980491 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.980489    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-var-lib-openvswitch\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:02.981078 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.980515    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f26cce38-f34e-4f48-9a02-88e6674fe266-system-cni-dir\") pod \"multus-additional-cni-plugins-zcv75\" (UID: \"f26cce38-f34e-4f48-9a02-88e6674fe266\") " pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:02.981078 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.980768    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5vcr5"
Sep 14 12:58:02.981310 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.981292    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 14 12:58:02.981379 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.981364    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 14 12:58:02.981589 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.981574    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 14 12:58:02.981790 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.981773    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8h7cx\""
Sep 14 12:58:02.981874 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.981815    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 14 12:58:02.982233 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.982217    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 14 12:58:02.983574 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.983555    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:02.985077 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.984557    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h8mlx\""
Sep 14 12:58:02.985077 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.984751    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 14 12:58:02.985077 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.984845    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 14 12:58:02.985077 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.984985    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 14 12:58:02.986558 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.986484    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 14 12:58:02.986698 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.986675    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t6ffv\""
Sep 14 12:58:02.986778 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.986706    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nhvd2"
Sep 14 12:58:02.986778 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.986715    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 14 12:58:02.987884 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.987863    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 14 12:58:02.990177 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.990163    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cz95j"
Sep 14 12:58:02.992642 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.992598    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4xdjj"
Sep 14 12:58:02.993132 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.993113    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7ls5v\""
Sep 14 12:58:02.993282 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.993265    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qrtm6\""
Sep 14 12:58:02.993501 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.993484    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 14 12:58:02.993570 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.993554    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 14 12:58:02.993619 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.993554    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 14 12:58:02.995066 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.995039    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:02.995155 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:02.995110    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgj9g" podUID="7eb7a234-f263-457d-9f06-c895898890ac"
Sep 14 12:58:02.995497 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.995473    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 14 12:58:02.995754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.995720    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 14 12:58:02.995825 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.995767    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 14 12:58:02.996041 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.995965    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qd26v\""
Sep 14 12:58:02.997380 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.997363    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b97dp"
Sep 14 12:58:02.999800 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:02.999781    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.000676 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.000627    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-13 12:53:02 +0000 UTC" deadline="2028-06-01 15:22:22.84649792 +0000 UTC"
Sep 14 12:58:03.000676 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.000675    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15026h24m19.845826127s"
Sep 14 12:58:03.000787 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.000677    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-64cc5\""
Sep 14 12:58:03.000787 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.000689    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 14 12:58:03.000787 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.000728    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 14 12:58:03.002105 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.002087    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:03.002187 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:03.002148    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcjld" podUID="2fafcfb0-af2a-41f5-8742-e9679b873e89"
Sep 14 12:58:03.002598 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.002581    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 14 12:58:03.002800 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.002785    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 14 12:58:03.002908 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.002891    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 14 12:58:03.002908 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.002899    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-znsxx\""
Sep 14 12:58:03.014641 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.014622    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kjm9w"
Sep 14 12:58:03.023675 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.023658    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kjm9w"
Sep 14 12:58:03.066354 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:03.066152    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0633c83db9ecaf6a98b80b3d362c853b.slice/crio-745d9d84379c6db185d06c241bb9f2f05ba34cfdaa3dd3a9cfbc08304b517e58 WatchSource:0}: Error finding container 745d9d84379c6db185d06c241bb9f2f05ba34cfdaa3dd3a9cfbc08304b517e58: Status 404 returned error can't find the container with id 745d9d84379c6db185d06c241bb9f2f05ba34cfdaa3dd3a9cfbc08304b517e58
Sep 14 12:58:03.069970 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.069952    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 12:58:03.076848 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.076823    2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 14 12:58:03.080878 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.080859    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-host-cni-netd\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.080974 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.080890    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-etc-modprobe-d\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.080974 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.080922    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-etc-sysconfig\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.080974 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.080927    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-host-cni-netd\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.080974 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.080942    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-etc-sysctl-conf\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.080974 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.080961    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml4t2\" (UniqueName: \"kubernetes.io/projected/0310cfeb-4b51-428f-9dd1-4dad5bd0cb1c-kube-api-access-ml4t2\") pod \"iptables-alerter-4xdjj\" (UID: \"0310cfeb-4b51-428f-9dd1-4dad5bd0cb1c\") " pod="openshift-network-operator/iptables-alerter-4xdjj"
Sep 14 12:58:03.081249 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.080990    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-systemd-units\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.081249 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081021    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-multus-socket-dir-parent\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.081249 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081037    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-host-var-lib-kubelet\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.081249 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081052    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-host-run-multus-certs\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.081249 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081065    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7eb7a234-f263-457d-9f06-c895898890ac-dbus\") pod \"global-pull-secret-syncer-zgj9g\" (UID: \"7eb7a234-f263-457d-9f06-c895898890ac\") " pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:03.081249 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081070    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-systemd-units\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.081249 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081080    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62d953d6-3b86-4c95-9a47-9579d88993a7-ovnkube-config\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.081249 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081099    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-etc-systemd\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.081249 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081131    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-lib-modules\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.081249 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081145    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-system-cni-dir\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.081249 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081159    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-host-run-k8s-cni-cncf-io\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.081249 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081182    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-multus-conf-dir\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.081249 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081198    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e8eda6eb-1097-4a02-b60a-45eb7ddcc87e-agent-certs\") pod \"konnectivity-agent-b97dp\" (UID: \"e8eda6eb-1097-4a02-b60a-45eb7ddcc87e\") " pod="kube-system/konnectivity-agent-b97dp"
Sep 14 12:58:03.081249 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081212    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f-socket-dir\") pod \"aws-ebs-csi-driver-node-77qtv\" (UID: \"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.081249 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081228    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-run-ovn\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.081249 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081251    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62d953d6-3b86-4c95-9a47-9579d88993a7-ovn-node-metrics-cert\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.082063 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081275    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-etc-tuned\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.082063 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081282    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-run-ovn\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.082063 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081298    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k87k\" (UniqueName: \"kubernetes.io/projected/e41b38b6-a3f2-4093-977d-c44ada18fdcb-kube-api-access-9k87k\") pod \"node-resolver-nhvd2\" (UID: \"e41b38b6-a3f2-4093-977d-c44ada18fdcb\") " pod="openshift-dns/node-resolver-nhvd2"
Sep 14 12:58:03.082063 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081323    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f-device-dir\") pod \"aws-ebs-csi-driver-node-77qtv\" (UID: \"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.082063 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081347    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-host-run-netns\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.082063 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081374    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-var-lib-openvswitch\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.082063 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081400    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f26cce38-f34e-4f48-9a02-88e6674fe266-system-cni-dir\") pod \"multus-additional-cni-plugins-zcv75\" (UID: \"f26cce38-f34e-4f48-9a02-88e6674fe266\") " pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:03.082063 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081419    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-host-run-netns\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.082063 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081423    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-etc-sysctl-d\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.082063 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081447    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b4458034-12ab-40d0-93c8-77de00c6ffb7-serviceca\") pod \"node-ca-5vcr5\" (UID: \"b4458034-12ab-40d0-93c8-77de00c6ffb7\") " pod="openshift-image-registry/node-ca-5vcr5"
Sep 14 12:58:03.082063 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081442    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-var-lib-openvswitch\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.082063 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081451    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f26cce38-f34e-4f48-9a02-88e6674fe266-system-cni-dir\") pod \"multus-additional-cni-plugins-zcv75\" (UID: \"f26cce38-f34e-4f48-9a02-88e6674fe266\") " pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:03.082063 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081473    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6hfq\" (UniqueName: \"kubernetes.io/projected/898a4541-4e3f-4276-b037-ff179ac5fc52-kube-api-access-t6hfq\") pod \"network-metrics-daemon-nl2vh\" (UID: \"898a4541-4e3f-4276-b037-ff179ac5fc52\") " pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:03.082063 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081506    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.082063 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081559    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3416c159-66b8-4cc1-844a-856cf03e8bc8-multus-daemon-config\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.082063 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081578    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62d953d6-3b86-4c95-9a47-9579d88993a7-ovnkube-config\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.082063 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081601    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.083102 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081583    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-77qtv\" (UID: \"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.083102 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081662    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f-registration-dir\") pod \"aws-ebs-csi-driver-node-77qtv\" (UID: \"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.083102 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081682    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62d953d6-3b86-4c95-9a47-9579d88993a7-env-overrides\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.083102 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081709    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f26cce38-f34e-4f48-9a02-88e6674fe266-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zcv75\" (UID: \"f26cce38-f34e-4f48-9a02-88e6674fe266\") " pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:03.083102 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081732    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-host\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.083102 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081844    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-host-kubelet\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.083102 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081888    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-host-run-ovn-kubernetes\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.083102 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081903    2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 14 12:58:03.083102 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081929    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-host-kubelet\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.083102 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081917    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g564q\" (UniqueName: \"kubernetes.io/projected/62d953d6-3b86-4c95-9a47-9579d88993a7-kube-api-access-g564q\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.083102 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.081988    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-host-run-ovn-kubernetes\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.083102 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082004    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhxg6\" (UniqueName: \"kubernetes.io/projected/9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f-kube-api-access-zhxg6\") pod \"aws-ebs-csi-driver-node-77qtv\" (UID: \"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.083102 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082035    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f26cce38-f34e-4f48-9a02-88e6674fe266-cnibin\") pod \"multus-additional-cni-plugins-zcv75\" (UID: \"f26cce38-f34e-4f48-9a02-88e6674fe266\") " pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:03.083102 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082083    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-sys\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.083102 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082113    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7eb7a234-f263-457d-9f06-c895898890ac-kubelet-config\") pod \"global-pull-secret-syncer-zgj9g\" (UID: \"7eb7a234-f263-457d-9f06-c895898890ac\") " pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:03.083102 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082118    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f26cce38-f34e-4f48-9a02-88e6674fe266-cnibin\") pod \"multus-additional-cni-plugins-zcv75\" (UID: \"f26cce38-f34e-4f48-9a02-88e6674fe266\") " pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:03.083102 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082140    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f-sys-fs\") pod \"aws-ebs-csi-driver-node-77qtv\" (UID: \"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.084035 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082192    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-var-lib-kubelet\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.084035 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082191    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62d953d6-3b86-4c95-9a47-9579d88993a7-env-overrides\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.084035 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082219    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-hostroot\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.084035 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082272    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0310cfeb-4b51-428f-9dd1-4dad5bd0cb1c-iptables-alerter-script\") pod \"iptables-alerter-4xdjj\" (UID: \"0310cfeb-4b51-428f-9dd1-4dad5bd0cb1c\") " pod="openshift-network-operator/iptables-alerter-4xdjj"
Sep 14 12:58:03.084035 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082298    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62d953d6-3b86-4c95-9a47-9579d88993a7-ovnkube-script-lib\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.084035 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082314    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f26cce38-f34e-4f48-9a02-88e6674fe266-cni-binary-copy\") pod \"multus-additional-cni-plugins-zcv75\" (UID: \"f26cce38-f34e-4f48-9a02-88e6674fe266\") " pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:03.084035 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082335    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6snnh\" (UniqueName: \"kubernetes.io/projected/b4458034-12ab-40d0-93c8-77de00c6ffb7-kube-api-access-6snnh\") pod \"node-ca-5vcr5\" (UID: \"b4458034-12ab-40d0-93c8-77de00c6ffb7\") " pod="openshift-image-registry/node-ca-5vcr5"
Sep 14 12:58:03.084035 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082445    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f-etc-selinux\") pod \"aws-ebs-csi-driver-node-77qtv\" (UID: \"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.084035 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082481    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs\") pod \"network-metrics-daemon-nl2vh\" (UID: \"898a4541-4e3f-4276-b037-ff179ac5fc52\") " pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:03.084035 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082505    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-host-cni-bin\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.084035 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082532    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f26cce38-f34e-4f48-9a02-88e6674fe266-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zcv75\" (UID: \"f26cce38-f34e-4f48-9a02-88e6674fe266\") " pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:03.084035 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082558    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv7zn\" (UniqueName: \"kubernetes.io/projected/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-kube-api-access-nv7zn\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.084035 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082585    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-host-var-lib-cni-bin\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.084035 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082601    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-host-cni-bin\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.084035 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082611    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-host-var-lib-cni-multus\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.084035 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:03.082635    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 12:58:03.084835 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:03.082702    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs podName:898a4541-4e3f-4276-b037-ff179ac5fc52 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:03.582673229 +0000 UTC m=+2.017348742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs") pod "network-metrics-daemon-nl2vh" (UID: "898a4541-4e3f-4276-b037-ff179ac5fc52") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 12:58:03.084835 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082636    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-etc-kubernetes\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.084835 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082738    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmxhx\" (UniqueName: \"kubernetes.io/projected/3416c159-66b8-4cc1-844a-856cf03e8bc8-kube-api-access-fmxhx\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.084835 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082797    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f26cce38-f34e-4f48-9a02-88e6674fe266-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zcv75\" (UID: \"f26cce38-f34e-4f48-9a02-88e6674fe266\") " pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:03.084835 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082808    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0310cfeb-4b51-428f-9dd1-4dad5bd0cb1c-host-slash\") pod \"iptables-alerter-4xdjj\" (UID: \"0310cfeb-4b51-428f-9dd1-4dad5bd0cb1c\") " pod="openshift-network-operator/iptables-alerter-4xdjj"
Sep 14 12:58:03.084835 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082881    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e41b38b6-a3f2-4093-977d-c44ada18fdcb-hosts-file\") pod \"node-resolver-nhvd2\" (UID: \"e41b38b6-a3f2-4093-977d-c44ada18fdcb\") " pod="openshift-dns/node-resolver-nhvd2"
Sep 14 12:58:03.084835 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082883    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62d953d6-3b86-4c95-9a47-9579d88993a7-ovnkube-script-lib\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.084835 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082930    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lbtl\" (UniqueName: \"kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl\") pod \"network-check-target-jcjld\" (UID: \"2fafcfb0-af2a-41f5-8742-e9679b873e89\") " pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:03.084835 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.082968    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-host-slash\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.084835 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083014    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-node-log\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.084835 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083038    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-tmp\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.084835 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083061    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-cnibin\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.084835 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083108    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret\") pod \"global-pull-secret-syncer-zgj9g\" (UID: \"7eb7a234-f263-457d-9f06-c895898890ac\") " pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:03.084835 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083134    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b4458034-12ab-40d0-93c8-77de00c6ffb7-host\") pod \"node-ca-5vcr5\" (UID: \"b4458034-12ab-40d0-93c8-77de00c6ffb7\") " pod="openshift-image-registry/node-ca-5vcr5"
Sep 14 12:58:03.084835 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083166    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-host-slash\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.084835 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083176    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-node-log\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.085687 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083175    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-run-systemd\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.085687 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083217    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-etc-openvswitch\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.085687 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083240    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-run-systemd\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.085687 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083247    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-log-socket\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.085687 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083255    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-etc-openvswitch\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.085687 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083279    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-etc-kubernetes\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.085687 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083300    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-log-socket\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.085687 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083333    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-run\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.085687 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083358    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-multus-cni-dir\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.085687 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083397    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3416c159-66b8-4cc1-844a-856cf03e8bc8-cni-binary-copy\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.085687 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083435    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-run-openvswitch\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.085687 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083462    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f26cce38-f34e-4f48-9a02-88e6674fe266-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zcv75\" (UID: \"f26cce38-f34e-4f48-9a02-88e6674fe266\") " pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:03.085687 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083503    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhlrp\" (UniqueName: \"kubernetes.io/projected/f26cce38-f34e-4f48-9a02-88e6674fe266-kube-api-access-vhlrp\") pod \"multus-additional-cni-plugins-zcv75\" (UID: \"f26cce38-f34e-4f48-9a02-88e6674fe266\") " pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:03.085687 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083530    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-host-run-netns\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.085687 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083556    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41b38b6-a3f2-4093-977d-c44ada18fdcb-tmp-dir\") pod \"node-resolver-nhvd2\" (UID: \"e41b38b6-a3f2-4093-977d-c44ada18fdcb\") " pod="openshift-dns/node-resolver-nhvd2"
Sep 14 12:58:03.085687 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083582    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e8eda6eb-1097-4a02-b60a-45eb7ddcc87e-konnectivity-ca\") pod \"konnectivity-agent-b97dp\" (UID: \"e8eda6eb-1097-4a02-b60a-45eb7ddcc87e\") " pod="kube-system/konnectivity-agent-b97dp"
Sep 14 12:58:03.085687 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083686    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f26cce38-f34e-4f48-9a02-88e6674fe266-cni-binary-copy\") pod \"multus-additional-cni-plugins-zcv75\" (UID: \"f26cce38-f34e-4f48-9a02-88e6674fe266\") " pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:03.086574 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083763    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62d953d6-3b86-4c95-9a47-9579d88993a7-run-openvswitch\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.086574 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.083824    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f26cce38-f34e-4f48-9a02-88e6674fe266-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zcv75\" (UID: \"f26cce38-f34e-4f48-9a02-88e6674fe266\") " pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:03.086574 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.084080    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f26cce38-f34e-4f48-9a02-88e6674fe266-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zcv75\" (UID: \"f26cce38-f34e-4f48-9a02-88e6674fe266\") " pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:03.086574 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.086224    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62d953d6-3b86-4c95-9a47-9579d88993a7-ovn-node-metrics-cert\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.090566 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.090539    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g564q\" (UniqueName: \"kubernetes.io/projected/62d953d6-3b86-4c95-9a47-9579d88993a7-kube-api-access-g564q\") pod \"ovnkube-node-jwqnf\" (UID: \"62d953d6-3b86-4c95-9a47-9579d88993a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.090662 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.090608    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6hfq\" (UniqueName: \"kubernetes.io/projected/898a4541-4e3f-4276-b037-ff179ac5fc52-kube-api-access-t6hfq\") pod \"network-metrics-daemon-nl2vh\" (UID: \"898a4541-4e3f-4276-b037-ff179ac5fc52\") " pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:03.091800 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.091781    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhlrp\" (UniqueName: \"kubernetes.io/projected/f26cce38-f34e-4f48-9a02-88e6674fe266-kube-api-access-vhlrp\") pod \"multus-additional-cni-plugins-zcv75\" (UID: \"f26cce38-f34e-4f48-9a02-88e6674fe266\") " pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:03.184712 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.184691    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhxg6\" (UniqueName: \"kubernetes.io/projected/9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f-kube-api-access-zhxg6\") pod \"aws-ebs-csi-driver-node-77qtv\" (UID: \"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.184834 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.184716    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-sys\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.184834 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.184736    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7eb7a234-f263-457d-9f06-c895898890ac-kubelet-config\") pod \"global-pull-secret-syncer-zgj9g\" (UID: \"7eb7a234-f263-457d-9f06-c895898890ac\") " pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:03.184834 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.184775    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f-sys-fs\") pod \"aws-ebs-csi-driver-node-77qtv\" (UID: \"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.184834 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.184793    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-var-lib-kubelet\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.184834 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.184811    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-hostroot\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.184834 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.184823    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-sys\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.185174 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.184835    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7eb7a234-f263-457d-9f06-c895898890ac-kubelet-config\") pod \"global-pull-secret-syncer-zgj9g\" (UID: \"7eb7a234-f263-457d-9f06-c895898890ac\") " pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:03.185174 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.184858    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f-sys-fs\") pod \"aws-ebs-csi-driver-node-77qtv\" (UID: \"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.185174 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.184868    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-hostroot\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.185174 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.184880    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0310cfeb-4b51-428f-9dd1-4dad5bd0cb1c-iptables-alerter-script\") pod \"iptables-alerter-4xdjj\" (UID: \"0310cfeb-4b51-428f-9dd1-4dad5bd0cb1c\") " pod="openshift-network-operator/iptables-alerter-4xdjj"
Sep 14 12:58:03.185174 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.184882    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-var-lib-kubelet\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.185174 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.184955    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6snnh\" (UniqueName: \"kubernetes.io/projected/b4458034-12ab-40d0-93c8-77de00c6ffb7-kube-api-access-6snnh\") pod \"node-ca-5vcr5\" (UID: \"b4458034-12ab-40d0-93c8-77de00c6ffb7\") " pod="openshift-image-registry/node-ca-5vcr5"
Sep 14 12:58:03.185174 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.184985    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f-etc-selinux\") pod \"aws-ebs-csi-driver-node-77qtv\" (UID: \"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.185174 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185050    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv7zn\" (UniqueName: \"kubernetes.io/projected/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-kube-api-access-nv7zn\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.185174 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185068    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-host-var-lib-cni-bin\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.185174 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185071    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f-etc-selinux\") pod \"aws-ebs-csi-driver-node-77qtv\" (UID: \"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.185174 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185085    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-host-var-lib-cni-multus\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.185174 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185107    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-etc-kubernetes\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.185174 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185125    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmxhx\" (UniqueName: \"kubernetes.io/projected/3416c159-66b8-4cc1-844a-856cf03e8bc8-kube-api-access-fmxhx\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.185174 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185130    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-host-var-lib-cni-multus\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.185174 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185111    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-host-var-lib-cni-bin\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.185174 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185165    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-etc-kubernetes\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.186010 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185199    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0310cfeb-4b51-428f-9dd1-4dad5bd0cb1c-host-slash\") pod \"iptables-alerter-4xdjj\" (UID: \"0310cfeb-4b51-428f-9dd1-4dad5bd0cb1c\") " pod="openshift-network-operator/iptables-alerter-4xdjj"
Sep 14 12:58:03.186010 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185228    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e41b38b6-a3f2-4093-977d-c44ada18fdcb-hosts-file\") pod \"node-resolver-nhvd2\" (UID: \"e41b38b6-a3f2-4093-977d-c44ada18fdcb\") " pod="openshift-dns/node-resolver-nhvd2"
Sep 14 12:58:03.186010 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185251    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0310cfeb-4b51-428f-9dd1-4dad5bd0cb1c-host-slash\") pod \"iptables-alerter-4xdjj\" (UID: \"0310cfeb-4b51-428f-9dd1-4dad5bd0cb1c\") " pod="openshift-network-operator/iptables-alerter-4xdjj"
Sep 14 12:58:03.186010 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185254    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbtl\" (UniqueName: \"kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl\") pod \"network-check-target-jcjld\" (UID: \"2fafcfb0-af2a-41f5-8742-e9679b873e89\") " pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:03.186010 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185284    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-tmp\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.186010 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185298    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e41b38b6-a3f2-4093-977d-c44ada18fdcb-hosts-file\") pod \"node-resolver-nhvd2\" (UID: \"e41b38b6-a3f2-4093-977d-c44ada18fdcb\") " pod="openshift-dns/node-resolver-nhvd2"
Sep 14 12:58:03.186010 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185316    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-cnibin\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.186010 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185377    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret\") pod \"global-pull-secret-syncer-zgj9g\" (UID: \"7eb7a234-f263-457d-9f06-c895898890ac\") " pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:03.186010 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185403    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b4458034-12ab-40d0-93c8-77de00c6ffb7-host\") pod \"node-ca-5vcr5\" (UID: \"b4458034-12ab-40d0-93c8-77de00c6ffb7\") " pod="openshift-image-registry/node-ca-5vcr5"
Sep 14 12:58:03.186010 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185431    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b4458034-12ab-40d0-93c8-77de00c6ffb7-host\") pod \"node-ca-5vcr5\" (UID: \"b4458034-12ab-40d0-93c8-77de00c6ffb7\") " pod="openshift-image-registry/node-ca-5vcr5"
Sep 14 12:58:03.186010 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185432    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-etc-kubernetes\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.186010 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185463    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-run\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.186010 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185465    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0310cfeb-4b51-428f-9dd1-4dad5bd0cb1c-iptables-alerter-script\") pod \"iptables-alerter-4xdjj\" (UID: \"0310cfeb-4b51-428f-9dd1-4dad5bd0cb1c\") " pod="openshift-network-operator/iptables-alerter-4xdjj"
Sep 14 12:58:03.186010 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185407    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-cnibin\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.186010 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:03.185468    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 12:58:03.186010 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185505    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-multus-cni-dir\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.186010 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185523    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3416c159-66b8-4cc1-844a-856cf03e8bc8-cni-binary-copy\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.186010 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185535    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-etc-kubernetes\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.186010 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185548    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-run\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.187012 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:03.185553    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret podName:7eb7a234-f263-457d-9f06-c895898890ac nodeName:}" failed. No retries permitted until 2026-09-14 12:58:03.685535244 +0000 UTC m=+2.120210744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret") pod "global-pull-secret-syncer-zgj9g" (UID: "7eb7a234-f263-457d-9f06-c895898890ac") : object "kube-system"/"original-pull-secret" not registered
Sep 14 12:58:03.187012 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185604    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-host-run-netns\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.187012 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185627    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41b38b6-a3f2-4093-977d-c44ada18fdcb-tmp-dir\") pod \"node-resolver-nhvd2\" (UID: \"e41b38b6-a3f2-4093-977d-c44ada18fdcb\") " pod="openshift-dns/node-resolver-nhvd2"
Sep 14 12:58:03.187012 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185651    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-multus-cni-dir\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.187012 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185674    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-host-run-netns\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.187012 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185698    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e8eda6eb-1097-4a02-b60a-45eb7ddcc87e-konnectivity-ca\") pod \"konnectivity-agent-b97dp\" (UID: \"e8eda6eb-1097-4a02-b60a-45eb7ddcc87e\") " pod="kube-system/konnectivity-agent-b97dp"
Sep 14 12:58:03.187012 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185726    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-etc-modprobe-d\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.187012 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185775    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-etc-sysconfig\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.187012 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185801    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-etc-sysctl-conf\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.187012 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185826    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml4t2\" (UniqueName: \"kubernetes.io/projected/0310cfeb-4b51-428f-9dd1-4dad5bd0cb1c-kube-api-access-ml4t2\") pod \"iptables-alerter-4xdjj\" (UID: \"0310cfeb-4b51-428f-9dd1-4dad5bd0cb1c\") " pod="openshift-network-operator/iptables-alerter-4xdjj"
Sep 14 12:58:03.187012 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185851    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-multus-socket-dir-parent\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.187012 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185854    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-etc-modprobe-d\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.187012 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185875    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-host-var-lib-kubelet\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.187012 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185899    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-host-run-multus-certs\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.187012 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185939    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41b38b6-a3f2-4093-977d-c44ada18fdcb-tmp-dir\") pod \"node-resolver-nhvd2\" (UID: \"e41b38b6-a3f2-4093-977d-c44ada18fdcb\") " pod="openshift-dns/node-resolver-nhvd2"
Sep 14 12:58:03.187012 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185925    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7eb7a234-f263-457d-9f06-c895898890ac-dbus\") pod \"global-pull-secret-syncer-zgj9g\" (UID: \"7eb7a234-f263-457d-9f06-c895898890ac\") " pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:03.187012 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185968    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-etc-systemd\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.187929 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.185993    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-lib-modules\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.187929 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186002    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-host-var-lib-kubelet\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.187929 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186015    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-etc-sysctl-conf\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.187929 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186051    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-multus-socket-dir-parent\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.187929 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186019    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-system-cni-dir\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.187929 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186074    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3416c159-66b8-4cc1-844a-856cf03e8bc8-cni-binary-copy\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.187929 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186097    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-etc-systemd\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.187929 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186078    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-system-cni-dir\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.187929 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186062    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-host-run-multus-certs\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.187929 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186096    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-host-run-k8s-cni-cncf-io\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.187929 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186129    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7eb7a234-f263-457d-9f06-c895898890ac-dbus\") pod \"global-pull-secret-syncer-zgj9g\" (UID: \"7eb7a234-f263-457d-9f06-c895898890ac\") " pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:03.187929 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186138    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-lib-modules\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.187929 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186138    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-host-run-k8s-cni-cncf-io\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.187929 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186151    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-multus-conf-dir\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.187929 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186167    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-etc-sysconfig\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.187929 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186180    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e8eda6eb-1097-4a02-b60a-45eb7ddcc87e-agent-certs\") pod \"konnectivity-agent-b97dp\" (UID: \"e8eda6eb-1097-4a02-b60a-45eb7ddcc87e\") " pod="kube-system/konnectivity-agent-b97dp"
Sep 14 12:58:03.187929 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186207    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f-socket-dir\") pod \"aws-ebs-csi-driver-node-77qtv\" (UID: \"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.187929 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186207    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3416c159-66b8-4cc1-844a-856cf03e8bc8-multus-conf-dir\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.188729 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186245    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-etc-tuned\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.188729 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186251    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e8eda6eb-1097-4a02-b60a-45eb7ddcc87e-konnectivity-ca\") pod \"konnectivity-agent-b97dp\" (UID: \"e8eda6eb-1097-4a02-b60a-45eb7ddcc87e\") " pod="kube-system/konnectivity-agent-b97dp"
Sep 14 12:58:03.188729 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186311    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f-socket-dir\") pod \"aws-ebs-csi-driver-node-77qtv\" (UID: \"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.188729 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186326    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k87k\" (UniqueName: \"kubernetes.io/projected/e41b38b6-a3f2-4093-977d-c44ada18fdcb-kube-api-access-9k87k\") pod \"node-resolver-nhvd2\" (UID: \"e41b38b6-a3f2-4093-977d-c44ada18fdcb\") " pod="openshift-dns/node-resolver-nhvd2"
Sep 14 12:58:03.188729 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186358    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f-device-dir\") pod \"aws-ebs-csi-driver-node-77qtv\" (UID: \"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.188729 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186402    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-etc-sysctl-d\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.188729 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186429    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b4458034-12ab-40d0-93c8-77de00c6ffb7-serviceca\") pod \"node-ca-5vcr5\" (UID: \"b4458034-12ab-40d0-93c8-77de00c6ffb7\") " pod="openshift-image-registry/node-ca-5vcr5"
Sep 14 12:58:03.188729 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186451    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f-device-dir\") pod \"aws-ebs-csi-driver-node-77qtv\" (UID: \"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.188729 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186457    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3416c159-66b8-4cc1-844a-856cf03e8bc8-multus-daemon-config\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.188729 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186483    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-77qtv\" (UID: \"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.188729 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186508    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f-registration-dir\") pod \"aws-ebs-csi-driver-node-77qtv\" (UID: \"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.188729 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186532    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-host\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.188729 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186564    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-etc-sysctl-d\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.188729 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186613    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-host\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.188729 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186615    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-77qtv\" (UID: \"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.188729 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186667    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f-registration-dir\") pod \"aws-ebs-csi-driver-node-77qtv\" (UID: \"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.188729 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186912    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b4458034-12ab-40d0-93c8-77de00c6ffb7-serviceca\") pod \"node-ca-5vcr5\" (UID: \"b4458034-12ab-40d0-93c8-77de00c6ffb7\") " pod="openshift-image-registry/node-ca-5vcr5"
Sep 14 12:58:03.189323 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.186983    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3416c159-66b8-4cc1-844a-856cf03e8bc8-multus-daemon-config\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.189323 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.187637    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-tmp\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.189323 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.188314    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-etc-tuned\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.189323 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.188842    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e8eda6eb-1097-4a02-b60a-45eb7ddcc87e-agent-certs\") pod \"konnectivity-agent-b97dp\" (UID: \"e8eda6eb-1097-4a02-b60a-45eb7ddcc87e\") " pod="kube-system/konnectivity-agent-b97dp"
Sep 14 12:58:03.194193 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:03.194166    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 12:58:03.194300 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:03.194198    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 12:58:03.194300 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:03.194212    2557 projected.go:196] Error preparing data for projected volume kube-api-access-7lbtl for pod openshift-network-diagnostics/network-check-target-jcjld: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 12:58:03.194300 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:03.194279    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl podName:2fafcfb0-af2a-41f5-8742-e9679b873e89 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:03.694262549 +0000 UTC m=+2.128938053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7lbtl" (UniqueName: "kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl") pod "network-check-target-jcjld" (UID: "2fafcfb0-af2a-41f5-8742-e9679b873e89") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 12:58:03.196044 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.196021    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k87k\" (UniqueName: \"kubernetes.io/projected/e41b38b6-a3f2-4093-977d-c44ada18fdcb-kube-api-access-9k87k\") pod \"node-resolver-nhvd2\" (UID: \"e41b38b6-a3f2-4093-977d-c44ada18fdcb\") " pod="openshift-dns/node-resolver-nhvd2"
Sep 14 12:58:03.196157 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.196117    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml4t2\" (UniqueName: \"kubernetes.io/projected/0310cfeb-4b51-428f-9dd1-4dad5bd0cb1c-kube-api-access-ml4t2\") pod \"iptables-alerter-4xdjj\" (UID: \"0310cfeb-4b51-428f-9dd1-4dad5bd0cb1c\") " pod="openshift-network-operator/iptables-alerter-4xdjj"
Sep 14 12:58:03.196431 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.196404    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmxhx\" (UniqueName: \"kubernetes.io/projected/3416c159-66b8-4cc1-844a-856cf03e8bc8-kube-api-access-fmxhx\") pod \"multus-cz95j\" (UID: \"3416c159-66b8-4cc1-844a-856cf03e8bc8\") " pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.196497 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.196478    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv7zn\" (UniqueName: \"kubernetes.io/projected/cda6ae89-5afa-4da8-8d8e-e46c163a79fb-kube-api-access-nv7zn\") pod \"tuned-xkznw\" (UID: \"cda6ae89-5afa-4da8-8d8e-e46c163a79fb\") " pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.196720 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.196705    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6snnh\" (UniqueName: \"kubernetes.io/projected/b4458034-12ab-40d0-93c8-77de00c6ffb7-kube-api-access-6snnh\") pod \"node-ca-5vcr5\" (UID: \"b4458034-12ab-40d0-93c8-77de00c6ffb7\") " pod="openshift-image-registry/node-ca-5vcr5"
Sep 14 12:58:03.196820 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.196802    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhxg6\" (UniqueName: \"kubernetes.io/projected/9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f-kube-api-access-zhxg6\") pod \"aws-ebs-csi-driver-node-77qtv\" (UID: \"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.293304 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.293248    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:03.303329 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:03.303310    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d953d6_3b86_4c95_9a47_9579d88993a7.slice/crio-54095224944b8d05a7fb0f960c4c7082bc7f09e5b4bb29a73f16e7e10a7bca1f WatchSource:0}: Error finding container 54095224944b8d05a7fb0f960c4c7082bc7f09e5b4bb29a73f16e7e10a7bca1f: Status 404 returned error can't find the container with id 54095224944b8d05a7fb0f960c4c7082bc7f09e5b4bb29a73f16e7e10a7bca1f
Sep 14 12:58:03.313484 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.313464    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zcv75"
Sep 14 12:58:03.321794 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:03.321770    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26cce38_f34e_4f48_9a02_88e6674fe266.slice/crio-08ae41e77812617a11d8fafa9dc67522b24d2ca6467788f233dc23c71df11cb2 WatchSource:0}: Error finding container 08ae41e77812617a11d8fafa9dc67522b24d2ca6467788f233dc23c71df11cb2: Status 404 returned error can't find the container with id 08ae41e77812617a11d8fafa9dc67522b24d2ca6467788f233dc23c71df11cb2
Sep 14 12:58:03.332522 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.332504    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5vcr5"
Sep 14 12:58:03.337551 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.337525    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xkznw"
Sep 14 12:58:03.341198 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:03.341177    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4458034_12ab_40d0_93c8_77de00c6ffb7.slice/crio-da0afba58dca142fa9eb0500c63022d024eff2363e42b293aabc7ffa4f614795 WatchSource:0}: Error finding container da0afba58dca142fa9eb0500c63022d024eff2363e42b293aabc7ffa4f614795: Status 404 returned error can't find the container with id da0afba58dca142fa9eb0500c63022d024eff2363e42b293aabc7ffa4f614795
Sep 14 12:58:03.341754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.341730    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nhvd2"
Sep 14 12:58:03.346225 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:03.346206    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda6ae89_5afa_4da8_8d8e_e46c163a79fb.slice/crio-d2f06cfd99e9283a5a05702cc4fc2fded7670215c3d3bebbc31bd254c9b9b926 WatchSource:0}: Error finding container d2f06cfd99e9283a5a05702cc4fc2fded7670215c3d3bebbc31bd254c9b9b926: Status 404 returned error can't find the container with id d2f06cfd99e9283a5a05702cc4fc2fded7670215c3d3bebbc31bd254c9b9b926
Sep 14 12:58:03.351078 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:03.351058    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41b38b6_a3f2_4093_977d_c44ada18fdcb.slice/crio-613558eff4978fa43337d95a4b17ae52215b793f3adb4f334e8403fd36108cfb WatchSource:0}: Error finding container 613558eff4978fa43337d95a4b17ae52215b793f3adb4f334e8403fd36108cfb: Status 404 returned error can't find the container with id 613558eff4978fa43337d95a4b17ae52215b793f3adb4f334e8403fd36108cfb
Sep 14 12:58:03.356809 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.356790    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cz95j"
Sep 14 12:58:03.366147 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:03.366125    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3416c159_66b8_4cc1_844a_856cf03e8bc8.slice/crio-65ec7ee4f1762ed2fbfc9bbc5b7d951eeef05f6ab764f4a8ae0da999bfdda5a2 WatchSource:0}: Error finding container 65ec7ee4f1762ed2fbfc9bbc5b7d951eeef05f6ab764f4a8ae0da999bfdda5a2: Status 404 returned error can't find the container with id 65ec7ee4f1762ed2fbfc9bbc5b7d951eeef05f6ab764f4a8ae0da999bfdda5a2
Sep 14 12:58:03.377025 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.377006    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4xdjj"
Sep 14 12:58:03.383977 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.383952    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b97dp"
Sep 14 12:58:03.384919 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:03.384900    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0310cfeb_4b51_428f_9dd1_4dad5bd0cb1c.slice/crio-b0f12f8169942df328d9ac803587c411e04429d0a4ea4af01df15f437eb8ff53 WatchSource:0}: Error finding container b0f12f8169942df328d9ac803587c411e04429d0a4ea4af01df15f437eb8ff53: Status 404 returned error can't find the container with id b0f12f8169942df328d9ac803587c411e04429d0a4ea4af01df15f437eb8ff53
Sep 14 12:58:03.392723 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:03.392701    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8eda6eb_1097_4a02_b60a_45eb7ddcc87e.slice/crio-c478f89c870a281ebaced1995ecb6c6e0bda44b4a9262049cfa147274cd6958f WatchSource:0}: Error finding container c478f89c870a281ebaced1995ecb6c6e0bda44b4a9262049cfa147274cd6958f: Status 404 returned error can't find the container with id c478f89c870a281ebaced1995ecb6c6e0bda44b4a9262049cfa147274cd6958f
Sep 14 12:58:03.397653 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.397595    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv"
Sep 14 12:58:03.407101 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:03.407080    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd8e2c3_7893_40ef_a7b3_ea421e9ffd4f.slice/crio-f96941f93a2bb108baa6e548ad9d5e4b3bc7f82bfe7c5a44630076daabc3022e WatchSource:0}: Error finding container f96941f93a2bb108baa6e548ad9d5e4b3bc7f82bfe7c5a44630076daabc3022e: Status 404 returned error can't find the container with id f96941f93a2bb108baa6e548ad9d5e4b3bc7f82bfe7c5a44630076daabc3022e
Sep 14 12:58:03.420555 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.420532    2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 12:58:03.589364 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.589312    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs\") pod \"network-metrics-daemon-nl2vh\" (UID: \"898a4541-4e3f-4276-b037-ff179ac5fc52\") " pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:03.589441 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:03.589402    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 12:58:03.589477 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:03.589445    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs podName:898a4541-4e3f-4276-b037-ff179ac5fc52 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:04.58943155 +0000 UTC m=+3.024107050 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs") pod "network-metrics-daemon-nl2vh" (UID: "898a4541-4e3f-4276-b037-ff179ac5fc52") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 12:58:03.689537 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.689515    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret\") pod \"global-pull-secret-syncer-zgj9g\" (UID: \"7eb7a234-f263-457d-9f06-c895898890ac\") " pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:03.689690 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:03.689669    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 12:58:03.689769 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:03.689736    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret podName:7eb7a234-f263-457d-9f06-c895898890ac nodeName:}" failed. No retries permitted until 2026-09-14 12:58:04.68971577 +0000 UTC m=+3.124391339 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret") pod "global-pull-secret-syncer-zgj9g" (UID: "7eb7a234-f263-457d-9f06-c895898890ac") : object "kube-system"/"original-pull-secret" not registered
Sep 14 12:58:03.789968 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:03.789934    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbtl\" (UniqueName: \"kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl\") pod \"network-check-target-jcjld\" (UID: \"2fafcfb0-af2a-41f5-8742-e9679b873e89\") " pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:03.790166 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:03.790139    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 12:58:03.790230 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:03.790169    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 12:58:03.790230 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:03.790183    2557 projected.go:196] Error preparing data for projected volume kube-api-access-7lbtl for pod openshift-network-diagnostics/network-check-target-jcjld: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 12:58:03.790340 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:03.790239    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl podName:2fafcfb0-af2a-41f5-8742-e9679b873e89 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:04.790220024 +0000 UTC m=+3.224895532 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lbtl" (UniqueName: "kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl") pod "network-check-target-jcjld" (UID: "2fafcfb0-af2a-41f5-8742-e9679b873e89") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 12:58:04.026728 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:04.026614    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 12:53:03 +0000 UTC" deadline="2028-06-18 17:50:00.032192578 +0000 UTC"
Sep 14 12:58:04.026728 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:04.026680    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15436h51m56.005517314s"
Sep 14 12:58:04.043018 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:04.042995    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:04.043166 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:04.043119    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgj9g" podUID="7eb7a234-f263-457d-9f06-c895898890ac"
Sep 14 12:58:04.076085 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:04.076030    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv" event={"ID":"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f","Type":"ContainerStarted","Data":"f96941f93a2bb108baa6e548ad9d5e4b3bc7f82bfe7c5a44630076daabc3022e"}
Sep 14 12:58:04.087919 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:04.087888    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cz95j" event={"ID":"3416c159-66b8-4cc1-844a-856cf03e8bc8","Type":"ContainerStarted","Data":"65ec7ee4f1762ed2fbfc9bbc5b7d951eeef05f6ab764f4a8ae0da999bfdda5a2"}
Sep 14 12:58:04.110707 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:04.110653    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nhvd2" event={"ID":"e41b38b6-a3f2-4093-977d-c44ada18fdcb","Type":"ContainerStarted","Data":"613558eff4978fa43337d95a4b17ae52215b793f3adb4f334e8403fd36108cfb"}
Sep 14 12:58:04.119664 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:04.119635    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xkznw" event={"ID":"cda6ae89-5afa-4da8-8d8e-e46c163a79fb","Type":"ContainerStarted","Data":"d2f06cfd99e9283a5a05702cc4fc2fded7670215c3d3bebbc31bd254c9b9b926"}
Sep 14 12:58:04.143955 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:04.143725    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5vcr5" event={"ID":"b4458034-12ab-40d0-93c8-77de00c6ffb7","Type":"ContainerStarted","Data":"da0afba58dca142fa9eb0500c63022d024eff2363e42b293aabc7ffa4f614795"}
Sep 14 12:58:04.183239 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:04.183208    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcv75" event={"ID":"f26cce38-f34e-4f48-9a02-88e6674fe266","Type":"ContainerStarted","Data":"08ae41e77812617a11d8fafa9dc67522b24d2ca6467788f233dc23c71df11cb2"}
Sep 14 12:58:04.201370 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:04.201133    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-61.ec2.internal" event={"ID":"f85c20039e12da5048d2ed5c7f69bdf5","Type":"ContainerStarted","Data":"3948297ee05b3e6c47bed514eb7925d0aa307c307587ff1db2234f040d5d698e"}
Sep 14 12:58:04.223774 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:04.223707    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-61.ec2.internal" event={"ID":"0633c83db9ecaf6a98b80b3d362c853b","Type":"ContainerStarted","Data":"745d9d84379c6db185d06c241bb9f2f05ba34cfdaa3dd3a9cfbc08304b517e58"}
Sep 14 12:58:04.229048 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:04.228942    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b97dp" event={"ID":"e8eda6eb-1097-4a02-b60a-45eb7ddcc87e","Type":"ContainerStarted","Data":"c478f89c870a281ebaced1995ecb6c6e0bda44b4a9262049cfa147274cd6958f"}
Sep 14 12:58:04.247974 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:04.247947    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4xdjj" event={"ID":"0310cfeb-4b51-428f-9dd1-4dad5bd0cb1c","Type":"ContainerStarted","Data":"b0f12f8169942df328d9ac803587c411e04429d0a4ea4af01df15f437eb8ff53"}
Sep 14 12:58:04.261924 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:04.261695    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf" event={"ID":"62d953d6-3b86-4c95-9a47-9579d88993a7","Type":"ContainerStarted","Data":"54095224944b8d05a7fb0f960c4c7082bc7f09e5b4bb29a73f16e7e10a7bca1f"}
Sep 14 12:58:04.354730 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:04.354661    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 12:58:04.597187 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:04.597117    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs\") pod \"network-metrics-daemon-nl2vh\" (UID: \"898a4541-4e3f-4276-b037-ff179ac5fc52\") " pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:04.597500 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:04.597316    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 12:58:04.597500 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:04.597382    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs podName:898a4541-4e3f-4276-b037-ff179ac5fc52 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:06.59736329 +0000 UTC m=+5.032038807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs") pod "network-metrics-daemon-nl2vh" (UID: "898a4541-4e3f-4276-b037-ff179ac5fc52") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 12:58:04.698882 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:04.697781    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret\") pod \"global-pull-secret-syncer-zgj9g\" (UID: \"7eb7a234-f263-457d-9f06-c895898890ac\") " pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:04.698882 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:04.697939    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 12:58:04.698882 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:04.698004    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret podName:7eb7a234-f263-457d-9f06-c895898890ac nodeName:}" failed. No retries permitted until 2026-09-14 12:58:06.69798486 +0000 UTC m=+5.132660366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret") pod "global-pull-secret-syncer-zgj9g" (UID: "7eb7a234-f263-457d-9f06-c895898890ac") : object "kube-system"/"original-pull-secret" not registered
Sep 14 12:58:04.799174 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:04.799115    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbtl\" (UniqueName: \"kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl\") pod \"network-check-target-jcjld\" (UID: \"2fafcfb0-af2a-41f5-8742-e9679b873e89\") " pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:04.799348 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:04.799304    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 12:58:04.799348 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:04.799328    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 12:58:04.799348 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:04.799340    2557 projected.go:196] Error preparing data for projected volume kube-api-access-7lbtl for pod openshift-network-diagnostics/network-check-target-jcjld: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 12:58:04.799522 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:04.799403    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl podName:2fafcfb0-af2a-41f5-8742-e9679b873e89 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:06.799376897 +0000 UTC m=+5.234052402 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lbtl" (UniqueName: "kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl") pod "network-check-target-jcjld" (UID: "2fafcfb0-af2a-41f5-8742-e9679b873e89") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 12:58:05.027743 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:05.027646    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 12:53:03 +0000 UTC" deadline="2028-04-15 00:18:04.248677569 +0000 UTC"
Sep 14 12:58:05.027743 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:05.027693    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13883h19m59.220990043s"
Sep 14 12:58:05.042259 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:05.042232    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:05.042547 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:05.042386    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nl2vh" podUID="898a4541-4e3f-4276-b037-ff179ac5fc52"
Sep 14 12:58:05.042978 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:05.042830    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:05.042978 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:05.042947    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcjld" podUID="2fafcfb0-af2a-41f5-8742-e9679b873e89"
Sep 14 12:58:05.261705 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:05.261601    2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 12:58:06.045433 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:06.045399    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:06.046028 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:06.045526    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgj9g" podUID="7eb7a234-f263-457d-9f06-c895898890ac"
Sep 14 12:58:06.617554 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:06.617424    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs\") pod \"network-metrics-daemon-nl2vh\" (UID: \"898a4541-4e3f-4276-b037-ff179ac5fc52\") " pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:06.617733 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:06.617598    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 12:58:06.617733 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:06.617661    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs podName:898a4541-4e3f-4276-b037-ff179ac5fc52 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:10.617642105 +0000 UTC m=+9.052317610 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs") pod "network-metrics-daemon-nl2vh" (UID: "898a4541-4e3f-4276-b037-ff179ac5fc52") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 12:58:06.717769 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:06.717720    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret\") pod \"global-pull-secret-syncer-zgj9g\" (UID: \"7eb7a234-f263-457d-9f06-c895898890ac\") " pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:06.717950 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:06.717863    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 12:58:06.717950 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:06.717914    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret podName:7eb7a234-f263-457d-9f06-c895898890ac nodeName:}" failed. No retries permitted until 2026-09-14 12:58:10.717900496 +0000 UTC m=+9.152575994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret") pod "global-pull-secret-syncer-zgj9g" (UID: "7eb7a234-f263-457d-9f06-c895898890ac") : object "kube-system"/"original-pull-secret" not registered
Sep 14 12:58:06.818941 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:06.818207    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbtl\" (UniqueName: \"kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl\") pod \"network-check-target-jcjld\" (UID: \"2fafcfb0-af2a-41f5-8742-e9679b873e89\") " pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:06.818941 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:06.818430    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 12:58:06.818941 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:06.818456    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 12:58:06.818941 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:06.818468    2557 projected.go:196] Error preparing data for projected volume kube-api-access-7lbtl for pod openshift-network-diagnostics/network-check-target-jcjld: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 12:58:06.818941 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:06.818527    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl podName:2fafcfb0-af2a-41f5-8742-e9679b873e89 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:10.818507825 +0000 UTC m=+9.253183339 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lbtl" (UniqueName: "kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl") pod "network-check-target-jcjld" (UID: "2fafcfb0-af2a-41f5-8742-e9679b873e89") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 12:58:07.043455 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:07.043075    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:07.043455 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:07.043226    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nl2vh" podUID="898a4541-4e3f-4276-b037-ff179ac5fc52"
Sep 14 12:58:07.043455 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:07.043298    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:07.043455 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:07.043370    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcjld" podUID="2fafcfb0-af2a-41f5-8742-e9679b873e89"
Sep 14 12:58:08.045100 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:08.044963    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:08.045752 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:08.045111    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgj9g" podUID="7eb7a234-f263-457d-9f06-c895898890ac"
Sep 14 12:58:09.042586 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:09.042545    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:09.042586 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:09.042591    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:09.042897 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:09.042704    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nl2vh" podUID="898a4541-4e3f-4276-b037-ff179ac5fc52"
Sep 14 12:58:09.043193 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:09.043166    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcjld" podUID="2fafcfb0-af2a-41f5-8742-e9679b873e89"
Sep 14 12:58:10.043200 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:10.042911    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:10.043200 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:10.043053    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgj9g" podUID="7eb7a234-f263-457d-9f06-c895898890ac"
Sep 14 12:58:10.652335 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:10.652292    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs\") pod \"network-metrics-daemon-nl2vh\" (UID: \"898a4541-4e3f-4276-b037-ff179ac5fc52\") " pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:10.652525 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:10.652488    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 12:58:10.652599 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:10.652560    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs podName:898a4541-4e3f-4276-b037-ff179ac5fc52 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:18.652539557 +0000 UTC m=+17.087215061 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs") pod "network-metrics-daemon-nl2vh" (UID: "898a4541-4e3f-4276-b037-ff179ac5fc52") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 12:58:10.753306 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:10.753270    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret\") pod \"global-pull-secret-syncer-zgj9g\" (UID: \"7eb7a234-f263-457d-9f06-c895898890ac\") " pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:10.753618 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:10.753481    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 12:58:10.753618 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:10.753559    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret podName:7eb7a234-f263-457d-9f06-c895898890ac nodeName:}" failed. No retries permitted until 2026-09-14 12:58:18.753537382 +0000 UTC m=+17.188212900 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret") pod "global-pull-secret-syncer-zgj9g" (UID: "7eb7a234-f263-457d-9f06-c895898890ac") : object "kube-system"/"original-pull-secret" not registered
Sep 14 12:58:10.854124 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:10.854033    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbtl\" (UniqueName: \"kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl\") pod \"network-check-target-jcjld\" (UID: \"2fafcfb0-af2a-41f5-8742-e9679b873e89\") " pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:10.854287 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:10.854220    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 12:58:10.854287 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:10.854254    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 12:58:10.854287 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:10.854269    2557 projected.go:196] Error preparing data for projected volume kube-api-access-7lbtl for pod openshift-network-diagnostics/network-check-target-jcjld: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 12:58:10.854453 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:10.854333    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl podName:2fafcfb0-af2a-41f5-8742-e9679b873e89 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:18.854314005 +0000 UTC m=+17.288989511 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lbtl" (UniqueName: "kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl") pod "network-check-target-jcjld" (UID: "2fafcfb0-af2a-41f5-8742-e9679b873e89") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 12:58:11.043271 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:11.043018    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:11.043271 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:11.043049    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:11.043271 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:11.043175    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nl2vh" podUID="898a4541-4e3f-4276-b037-ff179ac5fc52"
Sep 14 12:58:11.043911 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:11.043310    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcjld" podUID="2fafcfb0-af2a-41f5-8742-e9679b873e89"
Sep 14 12:58:12.047899 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:12.047777    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:12.048499 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:12.048463    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgj9g" podUID="7eb7a234-f263-457d-9f06-c895898890ac"
Sep 14 12:58:13.042734 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:13.042702    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:13.042909 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:13.042843    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nl2vh" podUID="898a4541-4e3f-4276-b037-ff179ac5fc52"
Sep 14 12:58:13.043272 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:13.043211    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:13.043402 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:13.043358    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcjld" podUID="2fafcfb0-af2a-41f5-8742-e9679b873e89"
Sep 14 12:58:14.043165 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:14.043135    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:14.043637 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:14.043270    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgj9g" podUID="7eb7a234-f263-457d-9f06-c895898890ac"
Sep 14 12:58:15.042911 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:15.042877    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:15.043047 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:15.042877    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:15.043047 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:15.043015    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcjld" podUID="2fafcfb0-af2a-41f5-8742-e9679b873e89"
Sep 14 12:58:15.043170 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:15.043150    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nl2vh" podUID="898a4541-4e3f-4276-b037-ff179ac5fc52"
Sep 14 12:58:16.042754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:16.042655    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:16.043180 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:16.042798    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgj9g" podUID="7eb7a234-f263-457d-9f06-c895898890ac"
Sep 14 12:58:17.042945 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:17.042904    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:17.043359 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:17.042907    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:17.043359 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:17.043034    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcjld" podUID="2fafcfb0-af2a-41f5-8742-e9679b873e89"
Sep 14 12:58:17.043359 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:17.043098    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nl2vh" podUID="898a4541-4e3f-4276-b037-ff179ac5fc52"
Sep 14 12:58:18.042364 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:18.042332    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:18.042523 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:18.042459    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgj9g" podUID="7eb7a234-f263-457d-9f06-c895898890ac"
Sep 14 12:58:18.715978 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:18.715946    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs\") pod \"network-metrics-daemon-nl2vh\" (UID: \"898a4541-4e3f-4276-b037-ff179ac5fc52\") " pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:18.716479 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:18.716070    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 12:58:18.716479 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:18.716123    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs podName:898a4541-4e3f-4276-b037-ff179ac5fc52 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:34.716108682 +0000 UTC m=+33.150784178 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs") pod "network-metrics-daemon-nl2vh" (UID: "898a4541-4e3f-4276-b037-ff179ac5fc52") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 12:58:18.817037 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:18.817002    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret\") pod \"global-pull-secret-syncer-zgj9g\" (UID: \"7eb7a234-f263-457d-9f06-c895898890ac\") " pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:18.817186 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:18.817166    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 12:58:18.817253 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:18.817239    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret podName:7eb7a234-f263-457d-9f06-c895898890ac nodeName:}" failed. No retries permitted until 2026-09-14 12:58:34.817218596 +0000 UTC m=+33.251894100 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret") pod "global-pull-secret-syncer-zgj9g" (UID: "7eb7a234-f263-457d-9f06-c895898890ac") : object "kube-system"/"original-pull-secret" not registered
Sep 14 12:58:18.918054 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:18.918032    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbtl\" (UniqueName: \"kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl\") pod \"network-check-target-jcjld\" (UID: \"2fafcfb0-af2a-41f5-8742-e9679b873e89\") " pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:18.918152 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:18.918142    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 12:58:18.918190 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:18.918162    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 12:58:18.918190 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:18.918171    2557 projected.go:196] Error preparing data for projected volume kube-api-access-7lbtl for pod openshift-network-diagnostics/network-check-target-jcjld: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 12:58:18.918259 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:18.918210    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl podName:2fafcfb0-af2a-41f5-8742-e9679b873e89 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:34.918199484 +0000 UTC m=+33.352874980 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lbtl" (UniqueName: "kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl") pod "network-check-target-jcjld" (UID: "2fafcfb0-af2a-41f5-8742-e9679b873e89") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 12:58:19.042365 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:19.042341    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:19.042449 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:19.042341    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:19.042545 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:19.042528    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nl2vh" podUID="898a4541-4e3f-4276-b037-ff179ac5fc52"
Sep 14 12:58:19.042579 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:19.042434    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcjld" podUID="2fafcfb0-af2a-41f5-8742-e9679b873e89"
Sep 14 12:58:20.043206 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:20.043169    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:20.043678 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:20.043290    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgj9g" podUID="7eb7a234-f263-457d-9f06-c895898890ac"
Sep 14 12:58:21.042565 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:21.042528    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:21.042565 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:21.042567    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:21.042804 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:21.042670    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nl2vh" podUID="898a4541-4e3f-4276-b037-ff179ac5fc52"
Sep 14 12:58:21.042849 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:21.042809    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcjld" podUID="2fafcfb0-af2a-41f5-8742-e9679b873e89"
Sep 14 12:58:22.043435 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:22.043400    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:22.043913 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:22.043534    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgj9g" podUID="7eb7a234-f263-457d-9f06-c895898890ac"
Sep 14 12:58:23.043322 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:23.043119    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:23.043464 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:23.043439    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nl2vh" podUID="898a4541-4e3f-4276-b037-ff179ac5fc52"
Sep 14 12:58:23.043855 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:23.043165    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:23.043950 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:23.043918    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcjld" podUID="2fafcfb0-af2a-41f5-8742-e9679b873e89"
Sep 14 12:58:23.315875 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:23.315843    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xkznw" event={"ID":"cda6ae89-5afa-4da8-8d8e-e46c163a79fb","Type":"ContainerStarted","Data":"b56e8213ee128a9c0c03ae6319babe010b37d7d14b72c9ab03759445d3401f62"}
Sep 14 12:58:23.317605 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:23.317574    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-61.ec2.internal" event={"ID":"f85c20039e12da5048d2ed5c7f69bdf5","Type":"ContainerStarted","Data":"dba2f609ffc104fa86548f84d4fbc05402cc487b3eceadbf16fa10cd0f52dc97"}
Sep 14 12:58:23.319673 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:23.319642    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf" event={"ID":"62d953d6-3b86-4c95-9a47-9579d88993a7","Type":"ContainerStarted","Data":"d43a4d638f94ae74bd253294f6967ef9e0d08cf71b3a6b91ed93228aca86ea28"}
Sep 14 12:58:23.319776 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:23.319723    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf" event={"ID":"62d953d6-3b86-4c95-9a47-9579d88993a7","Type":"ContainerStarted","Data":"eee76edfa8d8460da8b3230726d7a96a1e0f02a797d0fb01f4a40bbdbc0e400a"}
Sep 14 12:58:23.342619 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:23.342576    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xkznw" podStartSLOduration=1.851350304 podStartE2EDuration="21.342562504s" totalImagesPullingTime="19.4912122s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:23.342100087 +0000 UTC m=+21.776775606" watchObservedRunningTime="2026-09-14 12:58:23.342562504 +0000 UTC m=+21.777238043"
Sep 14 12:58:23.358646 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:23.358602    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-61.ec2.internal" podStartSLOduration=20.358586643 podStartE2EDuration="20.358586643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:23.358588396 +0000 UTC m=+21.793263915" watchObservedRunningTime="2026-09-14 12:58:23.358586643 +0000 UTC m=+21.793262143"
Sep 14 12:58:24.043347 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.043137    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:24.043492 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:24.043411    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgj9g" podUID="7eb7a234-f263-457d-9f06-c895898890ac"
Sep 14 12:58:24.323259 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.323186    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cz95j" event={"ID":"3416c159-66b8-4cc1-844a-856cf03e8bc8","Type":"ContainerStarted","Data":"5408f2beb775bcace290f760c7fe085fdf018b5a9649b8a3a97b233c4af214d0"}
Sep 14 12:58:24.325440 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.325412    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nhvd2" event={"ID":"e41b38b6-a3f2-4093-977d-c44ada18fdcb","Type":"ContainerStarted","Data":"d515c723bd9f1ead5e36c4ef43c279f36559a10dd2f769e4f6a87e2fcd3d00b5"}
Sep 14 12:58:24.330130 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.330105    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5vcr5" event={"ID":"b4458034-12ab-40d0-93c8-77de00c6ffb7","Type":"ContainerStarted","Data":"1da8f03ef1a148b526dc2d677ab16bbe6a9f2afcfd55e5a68e90de6d1c35aaaf"}
Sep 14 12:58:24.331900 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.331879    2557 generic.go:356] "Generic (PLEG): container finished" podID="f26cce38-f34e-4f48-9a02-88e6674fe266" containerID="0b186c987b55f4f42def2073d3476cb7bdf6a1e1863240ccd05c32e93642921a" exitCode=0
Sep 14 12:58:24.331994 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.331940    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcv75" event={"ID":"f26cce38-f34e-4f48-9a02-88e6674fe266","Type":"ContainerDied","Data":"0b186c987b55f4f42def2073d3476cb7bdf6a1e1863240ccd05c32e93642921a"}
Sep 14 12:58:24.333775 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.333753    2557 generic.go:356] "Generic (PLEG): container finished" podID="0633c83db9ecaf6a98b80b3d362c853b" containerID="05105cc96a1f602ab2f64a8e7919cf6e3d5d034a10a1911a1a64e296402e2422" exitCode=0
Sep 14 12:58:24.333907 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.333828    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-61.ec2.internal" event={"ID":"0633c83db9ecaf6a98b80b3d362c853b","Type":"ContainerDied","Data":"05105cc96a1f602ab2f64a8e7919cf6e3d5d034a10a1911a1a64e296402e2422"}
Sep 14 12:58:24.335620 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.335598    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b97dp" event={"ID":"e8eda6eb-1097-4a02-b60a-45eb7ddcc87e","Type":"ContainerStarted","Data":"0061a0ec02318a7dea35411aceacfec635346b5f7c62e846638d29a49b3bcaa7"}
Sep 14 12:58:24.337253 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.337230    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4xdjj" event={"ID":"0310cfeb-4b51-428f-9dd1-4dad5bd0cb1c","Type":"ContainerStarted","Data":"2c49b2a04ca62b8f2aa83bb225e8f22bbbca91c9fe87d83678bc667d104013d5"}
Sep 14 12:58:24.340386 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.340367    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwqnf_62d953d6-3b86-4c95-9a47-9579d88993a7/ovn-acl-logging/0.log"
Sep 14 12:58:24.340786 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.340763    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf" event={"ID":"62d953d6-3b86-4c95-9a47-9579d88993a7","Type":"ContainerDied","Data":"d43a4d638f94ae74bd253294f6967ef9e0d08cf71b3a6b91ed93228aca86ea28"}
Sep 14 12:58:24.340870 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.340785    2557 generic.go:356] "Generic (PLEG): container finished" podID="62d953d6-3b86-4c95-9a47-9579d88993a7" containerID="d43a4d638f94ae74bd253294f6967ef9e0d08cf71b3a6b91ed93228aca86ea28" exitCode=1
Sep 14 12:58:24.340870 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.340839    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf" event={"ID":"62d953d6-3b86-4c95-9a47-9579d88993a7","Type":"ContainerStarted","Data":"89b9218b62451e10ac181b5e5742cbae24c392fe3e69a2773be5fcf9b5831a51"}
Sep 14 12:58:24.340988 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.340889    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf" event={"ID":"62d953d6-3b86-4c95-9a47-9579d88993a7","Type":"ContainerStarted","Data":"591b1604d2b660e489e8a350ed813291dbaed89a0152b43a9f0d8bb94d864a37"}
Sep 14 12:58:24.341042 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.341013    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf" event={"ID":"62d953d6-3b86-4c95-9a47-9579d88993a7","Type":"ContainerStarted","Data":"7efb5fb3a183a34c09c9e33b8cc6bb8e0ba24e25db790247d0e9199406ba923e"}
Sep 14 12:58:24.341124 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.341065    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf" event={"ID":"62d953d6-3b86-4c95-9a47-9579d88993a7","Type":"ContainerStarted","Data":"e33fb19f2f07b6a71656be89339a51130c82ad420584a8f782ed4e083bab9317"}
Sep 14 12:58:24.342495 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.342476    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv" event={"ID":"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f","Type":"ContainerStarted","Data":"51b3c9792330eb062119c1ac4a6bbb99ab6075315f8c68a38f5b048ed4d3a2b8"}
Sep 14 12:58:24.343848 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.343806    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cz95j" podStartSLOduration=1.3351949140000001 podStartE2EDuration="21.343790734s" totalImagesPullingTime="20.00859582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:24.343500588 +0000 UTC m=+22.778176110" watchObservedRunningTime="2026-09-14 12:58:24.343790734 +0000 UTC m=+22.778466255"
Sep 14 12:58:24.359800 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.359765    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5vcr5" podStartSLOduration=2.908893561 podStartE2EDuration="22.35973453s" totalImagesPullingTime="19.450840969s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:24.359310219 +0000 UTC m=+22.793985739" watchObservedRunningTime="2026-09-14 12:58:24.35973453 +0000 UTC m=+22.794410048"
Sep 14 12:58:24.376949 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.376908    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b97dp" podStartSLOduration=1.97765602 podStartE2EDuration="21.376895073s" totalImagesPullingTime="19.399239053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:24.376681945 +0000 UTC m=+22.811357467" watchObservedRunningTime="2026-09-14 12:58:24.376895073 +0000 UTC m=+22.811570593"
Sep 14 12:58:24.443543 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.443486    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nhvd2" podStartSLOduration=1.9560159499999998 podStartE2EDuration="21.443463082s" totalImagesPullingTime="19.487447132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:24.442720778 +0000 UTC m=+22.877396298" watchObservedRunningTime="2026-09-14 12:58:24.443463082 +0000 UTC m=+22.878138602"
Sep 14 12:58:24.459153 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:24.459118    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4xdjj" podStartSLOduration=2.051968469 podStartE2EDuration="21.459107362s" totalImagesPullingTime="19.407138893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:24.459104656 +0000 UTC m=+22.893780174" watchObservedRunningTime="2026-09-14 12:58:24.459107362 +0000 UTC m=+22.893782881"
Sep 14 12:58:25.043206 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:25.043008    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:25.043353 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:25.043319    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcjld" podUID="2fafcfb0-af2a-41f5-8742-e9679b873e89"
Sep 14 12:58:25.043406 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:25.043357    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:25.043478 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:25.043461    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nl2vh" podUID="898a4541-4e3f-4276-b037-ff179ac5fc52"
Sep 14 12:58:25.052506 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:25.052485    2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 14 12:58:25.347057 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:25.346944    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-61.ec2.internal" event={"ID":"0633c83db9ecaf6a98b80b3d362c853b","Type":"ContainerStarted","Data":"8ef4e82cfb6b4f45bd1acaf36be8598a75bfb67d2051ab00b530dd9ec308155f"}
Sep 14 12:58:25.349028 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:25.348961    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv" event={"ID":"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f","Type":"ContainerStarted","Data":"5d702ac155b90aa3da5265b209740d01d597637eff9321005be59e80dc0393c4"}
Sep 14 12:58:25.362630 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:25.362585    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-61.ec2.internal" podStartSLOduration=22.362570924 podStartE2EDuration="22.362570924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:25.362309146 +0000 UTC m=+23.796984667" watchObservedRunningTime="2026-09-14 12:58:25.362570924 +0000 UTC m=+23.797246442"
Sep 14 12:58:26.042939 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:26.042422    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:26.042939 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:26.042547    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgj9g" podUID="7eb7a234-f263-457d-9f06-c895898890ac"
Sep 14 12:58:26.051879 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:26.051791    2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-14T12:58:25.052502811Z","UUID":"8a069a2d-3a7e-4fcd-8f03-c3564d2eab94","Handler":null,"Name":"","Endpoint":""}
Sep 14 12:58:26.053918 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:26.053893    2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 14 12:58:26.054295 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:26.053955    2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 14 12:58:26.354218 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:26.354198    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwqnf_62d953d6-3b86-4c95-9a47-9579d88993a7/ovn-acl-logging/0.log"
Sep 14 12:58:26.354683 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:26.354650    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf" event={"ID":"62d953d6-3b86-4c95-9a47-9579d88993a7","Type":"ContainerStarted","Data":"1382b5411d5f81aecca3c494a756fe050e470ed6147d476f8e483116c954f7da"}
Sep 14 12:58:27.042305 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:27.042276    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:27.042462 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:27.042374    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nl2vh" podUID="898a4541-4e3f-4276-b037-ff179ac5fc52"
Sep 14 12:58:27.042462 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:27.042429    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:27.042542 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:27.042505    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcjld" podUID="2fafcfb0-af2a-41f5-8742-e9679b873e89"
Sep 14 12:58:27.358733 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:27.358658    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv" event={"ID":"9cd8e2c3-7893-40ef-a7b3-ea421e9ffd4f","Type":"ContainerStarted","Data":"2c6a3876524ad0afc71c05efe8b8ff0fa58b09e92721a4edd3746ca1fcc8931f"}
Sep 14 12:58:27.377209 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:27.377155    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77qtv" podStartSLOduration=2.519949725 podStartE2EDuration="24.377138524s" totalImagesPullingTime="21.857188799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:58:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:27.376576381 +0000 UTC m=+25.811251903" watchObservedRunningTime="2026-09-14 12:58:27.377138524 +0000 UTC m=+25.811814044"
Sep 14 12:58:27.462064 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:27.462035    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b97dp"
Sep 14 12:58:27.462605 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:27.462573    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b97dp"
Sep 14 12:58:28.042541 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:28.042515    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:28.042717 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:28.042621    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgj9g" podUID="7eb7a234-f263-457d-9f06-c895898890ac"
Sep 14 12:58:28.361776 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:28.361721    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b97dp"
Sep 14 12:58:28.362418 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:28.362398    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b97dp"
Sep 14 12:58:29.043476 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:29.043257    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:29.043595 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:29.043281    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:29.043649 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:29.043597    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcjld" podUID="2fafcfb0-af2a-41f5-8742-e9679b873e89"
Sep 14 12:58:29.043698 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:29.043669    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nl2vh" podUID="898a4541-4e3f-4276-b037-ff179ac5fc52"
Sep 14 12:58:29.365847 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:29.365776    2557 generic.go:356] "Generic (PLEG): container finished" podID="f26cce38-f34e-4f48-9a02-88e6674fe266" containerID="8c8b7d5c0b09197a70e330dfb40fdb4e9be1fc9b679b168ff1c570166e35d633" exitCode=0
Sep 14 12:58:29.366190 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:29.365869    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcv75" event={"ID":"f26cce38-f34e-4f48-9a02-88e6674fe266","Type":"ContainerDied","Data":"8c8b7d5c0b09197a70e330dfb40fdb4e9be1fc9b679b168ff1c570166e35d633"}
Sep 14 12:58:29.372177 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:29.372161    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwqnf_62d953d6-3b86-4c95-9a47-9579d88993a7/ovn-acl-logging/0.log"
Sep 14 12:58:29.372571 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:29.372543    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf" event={"ID":"62d953d6-3b86-4c95-9a47-9579d88993a7","Type":"ContainerStarted","Data":"30510fc9c675880e51076ed170822e77b464d2a4ab4ff4bc30a16aa15931e579"}
Sep 14 12:58:29.372853 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:29.372835    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:29.372946 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:29.372901    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:29.373014 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:29.372958    2557 scope.go:117] "RemoveContainer" containerID="d43a4d638f94ae74bd253294f6967ef9e0d08cf71b3a6b91ed93228aca86ea28"
Sep 14 12:58:29.387897 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:29.387879    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:30.042983 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:30.042951    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:30.043249 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:30.043060    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgj9g" podUID="7eb7a234-f263-457d-9f06-c895898890ac"
Sep 14 12:58:30.379015 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:30.378950    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwqnf_62d953d6-3b86-4c95-9a47-9579d88993a7/ovn-acl-logging/0.log"
Sep 14 12:58:30.379605 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:30.379327    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf" event={"ID":"62d953d6-3b86-4c95-9a47-9579d88993a7","Type":"ContainerStarted","Data":"fc7c08e726791fc6c6e996ee3b1e4cba1796f3bd2fe4e4b51f7c0acac67fc850"}
Sep 14 12:58:30.379981 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:30.379752    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:30.395907 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:30.395883    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:58:30.442211 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:30.442159    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf" podStartSLOduration=8.725507233 podStartE2EDuration="28.442134028s" totalImagesPullingTime="19.716626795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:30.414314839 +0000 UTC m=+28.848990365" watchObservedRunningTime="2026-09-14 12:58:30.442134028 +0000 UTC m=+28.876809546"
Sep 14 12:58:30.660322 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:30.660292    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zgj9g"]
Sep 14 12:58:30.660459 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:30.660445    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:30.660594 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:30.660553    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgj9g" podUID="7eb7a234-f263-457d-9f06-c895898890ac"
Sep 14 12:58:30.663547 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:30.663526    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nl2vh"]
Sep 14 12:58:30.663648 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:30.663632    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:30.663788 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:30.663766    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nl2vh" podUID="898a4541-4e3f-4276-b037-ff179ac5fc52"
Sep 14 12:58:30.664219 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:30.664195    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jcjld"]
Sep 14 12:58:30.664319 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:30.664292    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:30.664380 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:30.664368    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcjld" podUID="2fafcfb0-af2a-41f5-8742-e9679b873e89"
Sep 14 12:58:31.385267 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:31.385228    2557 generic.go:356] "Generic (PLEG): container finished" podID="f26cce38-f34e-4f48-9a02-88e6674fe266" containerID="184f10d15c68fdff1f8a8938a048bb147514ab8063bdd54c3065a5c8d9decafe" exitCode=0
Sep 14 12:58:31.385885 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:31.385314    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcv75" event={"ID":"f26cce38-f34e-4f48-9a02-88e6674fe266","Type":"ContainerDied","Data":"184f10d15c68fdff1f8a8938a048bb147514ab8063bdd54c3065a5c8d9decafe"}
Sep 14 12:58:32.043348 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:32.043319    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:32.043525 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:32.043402    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:32.043525 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:32.043436    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:32.043525 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:32.043456    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgj9g" podUID="7eb7a234-f263-457d-9f06-c895898890ac"
Sep 14 12:58:32.043700 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:32.043540    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nl2vh" podUID="898a4541-4e3f-4276-b037-ff179ac5fc52"
Sep 14 12:58:32.043700 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:32.043594    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcjld" podUID="2fafcfb0-af2a-41f5-8742-e9679b873e89"
Sep 14 12:58:33.392693 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:33.392493    2557 generic.go:356] "Generic (PLEG): container finished" podID="f26cce38-f34e-4f48-9a02-88e6674fe266" containerID="93c9a16dad0f207d7b8ada401420c07cb33ee34b743d3621bab7acc45b551e92" exitCode=0
Sep 14 12:58:33.393035 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:33.392569    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcv75" event={"ID":"f26cce38-f34e-4f48-9a02-88e6674fe266","Type":"ContainerDied","Data":"93c9a16dad0f207d7b8ada401420c07cb33ee34b743d3621bab7acc45b551e92"}
Sep 14 12:58:34.042619 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:34.042595    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:34.042896 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:34.042601    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:34.042896 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:34.042711    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcjld" podUID="2fafcfb0-af2a-41f5-8742-e9679b873e89"
Sep 14 12:58:34.042896 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:34.042605    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:34.043073 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:34.042930    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nl2vh" podUID="898a4541-4e3f-4276-b037-ff179ac5fc52"
Sep 14 12:58:34.043073 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:34.043026    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgj9g" podUID="7eb7a234-f263-457d-9f06-c895898890ac"
Sep 14 12:58:34.745977 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:34.745893    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs\") pod \"network-metrics-daemon-nl2vh\" (UID: \"898a4541-4e3f-4276-b037-ff179ac5fc52\") " pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:34.746379 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:34.746039    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 12:58:34.746379 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:34.746108    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs podName:898a4541-4e3f-4276-b037-ff179ac5fc52 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:06.746088752 +0000 UTC m=+65.180764253 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs") pod "network-metrics-daemon-nl2vh" (UID: "898a4541-4e3f-4276-b037-ff179ac5fc52") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 12:58:34.847148 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:34.847114    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret\") pod \"global-pull-secret-syncer-zgj9g\" (UID: \"7eb7a234-f263-457d-9f06-c895898890ac\") " pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:34.847313 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:34.847231    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 12:58:34.847313 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:34.847282    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret podName:7eb7a234-f263-457d-9f06-c895898890ac nodeName:}" failed. No retries permitted until 2026-09-14 12:59:06.84726951 +0000 UTC m=+65.281945010 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret") pod "global-pull-secret-syncer-zgj9g" (UID: "7eb7a234-f263-457d-9f06-c895898890ac") : object "kube-system"/"original-pull-secret" not registered
Sep 14 12:58:34.947791 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:34.947753    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbtl\" (UniqueName: \"kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl\") pod \"network-check-target-jcjld\" (UID: \"2fafcfb0-af2a-41f5-8742-e9679b873e89\") " pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:34.947920 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:34.947864    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 12:58:34.947920 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:34.947886    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 12:58:34.947920 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:34.947897    2557 projected.go:196] Error preparing data for projected volume kube-api-access-7lbtl for pod openshift-network-diagnostics/network-check-target-jcjld: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 12:58:34.948069 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:34.947945    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl podName:2fafcfb0-af2a-41f5-8742-e9679b873e89 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:06.947929904 +0000 UTC m=+65.382605400 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lbtl" (UniqueName: "kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl") pod "network-check-target-jcjld" (UID: "2fafcfb0-af2a-41f5-8742-e9679b873e89") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 12:58:36.043339 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.043302    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:36.043689 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.043305    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:36.043689 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:36.043428    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcjld" podUID="2fafcfb0-af2a-41f5-8742-e9679b873e89"
Sep 14 12:58:36.043689 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.043459    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:36.043689 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:36.043587    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nl2vh" podUID="898a4541-4e3f-4276-b037-ff179ac5fc52"
Sep 14 12:58:36.043689 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:36.043682    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgj9g" podUID="7eb7a234-f263-457d-9f06-c895898890ac"
Sep 14 12:58:36.373254 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.373174    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-61.ec2.internal" event="NodeReady"
Sep 14 12:58:36.373421 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.373295    2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 14 12:58:36.407754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.407716    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"]
Sep 14 12:58:36.411690 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.411667    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8565875799-kvq4d"]
Sep 14 12:58:36.411838 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.411814    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"
Sep 14 12:58:36.415731 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.415714    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84458465cd-drbkw"]
Sep 14 12:58:36.415839 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.415824    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.419597 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.419576    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-795d86c4f-rnppz"]
Sep 14 12:58:36.420155 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.419698    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84458465cd-drbkw"
Sep 14 12:58:36.420155 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.420018    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 14 12:58:36.420323 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.420078    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 14 12:58:36.420426 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.420307    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 14 12:58:36.420426 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.420347    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 14 12:58:36.420426 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.420357    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 14 12:58:36.420562 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.420347    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 14 12:58:36.420942 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.420903    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 14 12:58:36.420942 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.420917    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 14 12:58:36.421085 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.420920    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 14 12:58:36.421085 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.420960    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 14 12:58:36.421085 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.421026    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8dx6q\""
Sep 14 12:58:36.423876 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.423856    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795d86c4f-rnppz"
Sep 14 12:58:36.424264 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.424247    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 14 12:58:36.425626 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.425593    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-p8jkc\""
Sep 14 12:58:36.426619 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.426600    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"]
Sep 14 12:58:36.428502 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.428444    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 14 12:58:36.428599 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.428542    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 14 12:58:36.431088 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.431066    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jmcw8"]
Sep 14 12:58:36.435023 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.434988    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gw4vc"]
Sep 14 12:58:36.435359 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.435346    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jmcw8"
Sep 14 12:58:36.439166 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.439142    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84458465cd-drbkw"]
Sep 14 12:58:36.439274 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.439186    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8565875799-kvq4d"]
Sep 14 12:58:36.439274 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.439261    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5hqfk\""
Sep 14 12:58:36.439366 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.439286    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:58:36.442257 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.439507    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 14 12:58:36.442257 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.439697    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 14 12:58:36.442257 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.440603    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 14 12:58:36.442257 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.442068    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 14 12:58:36.442257 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.442217    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xjstd\""
Sep 14 12:58:36.442939 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.442920    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 14 12:58:36.447754 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.447706    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gw4vc"]
Sep 14 12:58:36.450933 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.450911    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jmcw8"]
Sep 14 12:58:36.454566 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.454510    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-795d86c4f-rnppz"]
Sep 14 12:58:36.557914 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.557880    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2a94c6ba-2e5a-4dd5-9530-f146816e158c-hub\") pod \"cluster-proxy-proxy-agent-7df7df9c64-kzgms\" (UID: \"2a94c6ba-2e5a-4dd5-9530-f146816e158c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"
Sep 14 12:58:36.557914 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.557920    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls\") pod \"dns-default-gw4vc\" (UID: \"f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6\") " pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:58:36.558139 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.557945    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-installation-pull-secrets\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.558139 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.558055    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmx2q\" (UniqueName: \"kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-kube-api-access-hmx2q\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.558139 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.558105    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2a94c6ba-2e5a-4dd5-9530-f146816e158c-ca\") pod \"cluster-proxy-proxy-agent-7df7df9c64-kzgms\" (UID: \"2a94c6ba-2e5a-4dd5-9530-f146816e158c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"
Sep 14 12:58:36.558139 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.558128    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksj79\" (UniqueName: \"kubernetes.io/projected/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-kube-api-access-ksj79\") pod \"dns-default-gw4vc\" (UID: \"f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6\") " pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:58:36.558334 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.558155    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-certificates\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.558334 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.558181    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct65g\" (UniqueName: \"kubernetes.io/projected/ceb40da7-57e5-4547-bd7b-02d0c3c3e8ef-kube-api-access-ct65g\") pod \"managed-serviceaccount-addon-agent-84458465cd-drbkw\" (UID: \"ceb40da7-57e5-4547-bd7b-02d0c3c3e8ef\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84458465cd-drbkw"
Sep 14 12:58:36.558334 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.558205    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nld5n\" (UniqueName: \"kubernetes.io/projected/d44e13a7-7e76-4eee-873f-68bf4066ac29-kube-api-access-nld5n\") pod \"ingress-canary-jmcw8\" (UID: \"d44e13a7-7e76-4eee-873f-68bf4066ac29\") " pod="openshift-ingress-canary/ingress-canary-jmcw8"
Sep 14 12:58:36.558334 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.558231    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-config-volume\") pod \"dns-default-gw4vc\" (UID: \"f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6\") " pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:58:36.558334 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.558285    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ceb40da7-57e5-4547-bd7b-02d0c3c3e8ef-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-84458465cd-drbkw\" (UID: \"ceb40da7-57e5-4547-bd7b-02d0c3c3e8ef\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84458465cd-drbkw"
Sep 14 12:58:36.558334 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.558329    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a94c6ba-2e5a-4dd5-9530-f146816e158c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7df7df9c64-kzgms\" (UID: \"2a94c6ba-2e5a-4dd5-9530-f146816e158c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"
Sep 14 12:58:36.558623 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.558361    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-bound-sa-token\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.558623 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.558384    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d44e13a7-7e76-4eee-873f-68bf4066ac29-cert\") pod \"ingress-canary-jmcw8\" (UID: \"d44e13a7-7e76-4eee-873f-68bf4066ac29\") " pod="openshift-ingress-canary/ingress-canary-jmcw8"
Sep 14 12:58:36.558623 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.558453    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a94c6ba-2e5a-4dd5-9530-f146816e158c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7df7df9c64-kzgms\" (UID: \"2a94c6ba-2e5a-4dd5-9530-f146816e158c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"
Sep 14 12:58:36.558623 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.558476    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2a94c6ba-2e5a-4dd5-9530-f146816e158c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7df7df9c64-kzgms\" (UID: \"2a94c6ba-2e5a-4dd5-9530-f146816e158c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"
Sep 14 12:58:36.558623 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.558520    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q4b2\" (UniqueName: \"kubernetes.io/projected/2a94c6ba-2e5a-4dd5-9530-f146816e158c-kube-api-access-8q4b2\") pod \"cluster-proxy-proxy-agent-7df7df9c64-kzgms\" (UID: \"2a94c6ba-2e5a-4dd5-9530-f146816e158c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"
Sep 14 12:58:36.558623 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.558552    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/22dbba95-a3d2-4356-a088-55270cfa2f7a-klusterlet-config\") pod \"klusterlet-addon-workmgr-795d86c4f-rnppz\" (UID: \"22dbba95-a3d2-4356-a088-55270cfa2f7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795d86c4f-rnppz"
Sep 14 12:58:36.558623 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.558575    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-tmp-dir\") pod \"dns-default-gw4vc\" (UID: \"f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6\") " pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:58:36.558623 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.558595    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22dbba95-a3d2-4356-a088-55270cfa2f7a-tmp\") pod \"klusterlet-addon-workmgr-795d86c4f-rnppz\" (UID: \"22dbba95-a3d2-4356-a088-55270cfa2f7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795d86c4f-rnppz"
Sep 14 12:58:36.558623 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.558611    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfc9z\" (UniqueName: \"kubernetes.io/projected/22dbba95-a3d2-4356-a088-55270cfa2f7a-kube-api-access-bfc9z\") pod \"klusterlet-addon-workmgr-795d86c4f-rnppz\" (UID: \"22dbba95-a3d2-4356-a088-55270cfa2f7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795d86c4f-rnppz"
Sep 14 12:58:36.559046 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.558634    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.559046 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.558662    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-image-registry-private-configuration\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.559046 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.558687    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-ca-trust-extracted\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.559046 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.558712    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-trusted-ca\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.608202 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.608165    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b97dp_e8eda6eb-1097-4a02-b60a-45eb7ddcc87e/konnectivity-agent/0.log" limit=1024
Sep 14 12:58:36.659810 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.659714    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct65g\" (UniqueName: \"kubernetes.io/projected/ceb40da7-57e5-4547-bd7b-02d0c3c3e8ef-kube-api-access-ct65g\") pod \"managed-serviceaccount-addon-agent-84458465cd-drbkw\" (UID: \"ceb40da7-57e5-4547-bd7b-02d0c3c3e8ef\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84458465cd-drbkw"
Sep 14 12:58:36.659810 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.659765    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nld5n\" (UniqueName: \"kubernetes.io/projected/d44e13a7-7e76-4eee-873f-68bf4066ac29-kube-api-access-nld5n\") pod \"ingress-canary-jmcw8\" (UID: \"d44e13a7-7e76-4eee-873f-68bf4066ac29\") " pod="openshift-ingress-canary/ingress-canary-jmcw8"
Sep 14 12:58:36.659810 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.659793    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-config-volume\") pod \"dns-default-gw4vc\" (UID: \"f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6\") " pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:58:36.660029 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.659815    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ceb40da7-57e5-4547-bd7b-02d0c3c3e8ef-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-84458465cd-drbkw\" (UID: \"ceb40da7-57e5-4547-bd7b-02d0c3c3e8ef\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84458465cd-drbkw"
Sep 14 12:58:36.660029 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.659840    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a94c6ba-2e5a-4dd5-9530-f146816e158c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7df7df9c64-kzgms\" (UID: \"2a94c6ba-2e5a-4dd5-9530-f146816e158c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"
Sep 14 12:58:36.660029 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.659865    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-bound-sa-token\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.660162 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.660044    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d44e13a7-7e76-4eee-873f-68bf4066ac29-cert\") pod \"ingress-canary-jmcw8\" (UID: \"d44e13a7-7e76-4eee-873f-68bf4066ac29\") " pod="openshift-ingress-canary/ingress-canary-jmcw8"
Sep 14 12:58:36.660162 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.660110    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a94c6ba-2e5a-4dd5-9530-f146816e158c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7df7df9c64-kzgms\" (UID: \"2a94c6ba-2e5a-4dd5-9530-f146816e158c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"
Sep 14 12:58:36.660162 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.660136    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2a94c6ba-2e5a-4dd5-9530-f146816e158c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7df7df9c64-kzgms\" (UID: \"2a94c6ba-2e5a-4dd5-9530-f146816e158c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"
Sep 14 12:58:36.660300 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.660172    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q4b2\" (UniqueName: \"kubernetes.io/projected/2a94c6ba-2e5a-4dd5-9530-f146816e158c-kube-api-access-8q4b2\") pod \"cluster-proxy-proxy-agent-7df7df9c64-kzgms\" (UID: \"2a94c6ba-2e5a-4dd5-9530-f146816e158c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"
Sep 14 12:58:36.660487 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.660461    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/22dbba95-a3d2-4356-a088-55270cfa2f7a-klusterlet-config\") pod \"klusterlet-addon-workmgr-795d86c4f-rnppz\" (UID: \"22dbba95-a3d2-4356-a088-55270cfa2f7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795d86c4f-rnppz"
Sep 14 12:58:36.660556 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.660486    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-config-volume\") pod \"dns-default-gw4vc\" (UID: \"f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6\") " pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:58:36.660556 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.660502    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-tmp-dir\") pod \"dns-default-gw4vc\" (UID: \"f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6\") " pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:58:36.660556 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.660547    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22dbba95-a3d2-4356-a088-55270cfa2f7a-tmp\") pod \"klusterlet-addon-workmgr-795d86c4f-rnppz\" (UID: \"22dbba95-a3d2-4356-a088-55270cfa2f7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795d86c4f-rnppz"
Sep 14 12:58:36.660712 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.660575    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfc9z\" (UniqueName: \"kubernetes.io/projected/22dbba95-a3d2-4356-a088-55270cfa2f7a-kube-api-access-bfc9z\") pod \"klusterlet-addon-workmgr-795d86c4f-rnppz\" (UID: \"22dbba95-a3d2-4356-a088-55270cfa2f7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795d86c4f-rnppz"
Sep 14 12:58:36.660712 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.660599    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.660712 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.660630    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-image-registry-private-configuration\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.660712 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.660656    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-ca-trust-extracted\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.660712 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.660690    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-trusted-ca\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.660712 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.660699    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-tmp-dir\") pod \"dns-default-gw4vc\" (UID: \"f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6\") " pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:58:36.661025 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.660763    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2a94c6ba-2e5a-4dd5-9530-f146816e158c-hub\") pod \"cluster-proxy-proxy-agent-7df7df9c64-kzgms\" (UID: \"2a94c6ba-2e5a-4dd5-9530-f146816e158c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"
Sep 14 12:58:36.661025 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.660789    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls\") pod \"dns-default-gw4vc\" (UID: \"f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6\") " pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:58:36.661025 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.660813    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-installation-pull-secrets\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.661025 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.660844    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmx2q\" (UniqueName: \"kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-kube-api-access-hmx2q\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.661025 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.660853    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a94c6ba-2e5a-4dd5-9530-f146816e158c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7df7df9c64-kzgms\" (UID: \"2a94c6ba-2e5a-4dd5-9530-f146816e158c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"
Sep 14 12:58:36.661025 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.660874    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2a94c6ba-2e5a-4dd5-9530-f146816e158c-ca\") pod \"cluster-proxy-proxy-agent-7df7df9c64-kzgms\" (UID: \"2a94c6ba-2e5a-4dd5-9530-f146816e158c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"
Sep 14 12:58:36.661025 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.660899    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksj79\" (UniqueName: \"kubernetes.io/projected/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-kube-api-access-ksj79\") pod \"dns-default-gw4vc\" (UID: \"f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6\") " pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:58:36.661025 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.660930    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-certificates\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.661400 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.661083    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22dbba95-a3d2-4356-a088-55270cfa2f7a-tmp\") pod \"klusterlet-addon-workmgr-795d86c4f-rnppz\" (UID: \"22dbba95-a3d2-4356-a088-55270cfa2f7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795d86c4f-rnppz"
Sep 14 12:58:36.661804 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.661464    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-certificates\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.661804 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.661537    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-ca-trust-extracted\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.661804 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:36.660791    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 12:58:36.661804 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:36.661586    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8565875799-kvq4d: secret "image-registry-tls" not found
Sep 14 12:58:36.661804 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:36.661652    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls podName:f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e nodeName:}" failed. No retries permitted until 2026-09-14 12:58:37.161632761 +0000 UTC m=+35.596308258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls") pod "image-registry-8565875799-kvq4d" (UID: "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e") : secret "image-registry-tls" not found
Sep 14 12:58:36.662673 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.662637    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-trusted-ca\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.663167 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:36.661563    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 12:58:36.663167 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:36.662955    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls podName:f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:37.162937492 +0000 UTC m=+35.597612993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls") pod "dns-default-gw4vc" (UID: "f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6") : secret "dns-default-metrics-tls" not found
Sep 14 12:58:36.665217 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.665195    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a94c6ba-2e5a-4dd5-9530-f146816e158c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7df7df9c64-kzgms\" (UID: \"2a94c6ba-2e5a-4dd5-9530-f146816e158c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"
Sep 14 12:58:36.665217 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.665208    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2a94c6ba-2e5a-4dd5-9530-f146816e158c-ca\") pod \"cluster-proxy-proxy-agent-7df7df9c64-kzgms\" (UID: \"2a94c6ba-2e5a-4dd5-9530-f146816e158c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"
Sep 14 12:58:36.665362 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.665197    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-image-registry-private-configuration\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.665362 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.665230    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2a94c6ba-2e5a-4dd5-9530-f146816e158c-hub\") pod \"cluster-proxy-proxy-agent-7df7df9c64-kzgms\" (UID: \"2a94c6ba-2e5a-4dd5-9530-f146816e158c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"
Sep 14 12:58:36.665362 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.665250    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d44e13a7-7e76-4eee-873f-68bf4066ac29-cert\") pod \"ingress-canary-jmcw8\" (UID: \"d44e13a7-7e76-4eee-873f-68bf4066ac29\") " pod="openshift-ingress-canary/ingress-canary-jmcw8"
Sep 14 12:58:36.665672 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.665647    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2a94c6ba-2e5a-4dd5-9530-f146816e158c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7df7df9c64-kzgms\" (UID: \"2a94c6ba-2e5a-4dd5-9530-f146816e158c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"
Sep 14 12:58:36.665758 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.665687    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ceb40da7-57e5-4547-bd7b-02d0c3c3e8ef-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-84458465cd-drbkw\" (UID: \"ceb40da7-57e5-4547-bd7b-02d0c3c3e8ef\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84458465cd-drbkw"
Sep 14 12:58:36.666776 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.666729    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-installation-pull-secrets\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.667264 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.667233    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/22dbba95-a3d2-4356-a088-55270cfa2f7a-klusterlet-config\") pod \"klusterlet-addon-workmgr-795d86c4f-rnppz\" (UID: \"22dbba95-a3d2-4356-a088-55270cfa2f7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795d86c4f-rnppz"
Sep 14 12:58:36.669095 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.669065    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct65g\" (UniqueName: \"kubernetes.io/projected/ceb40da7-57e5-4547-bd7b-02d0c3c3e8ef-kube-api-access-ct65g\") pod \"managed-serviceaccount-addon-agent-84458465cd-drbkw\" (UID: \"ceb40da7-57e5-4547-bd7b-02d0c3c3e8ef\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84458465cd-drbkw"
Sep 14 12:58:36.669303 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.669285    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-bound-sa-token\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.669613 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.669593    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q4b2\" (UniqueName: \"kubernetes.io/projected/2a94c6ba-2e5a-4dd5-9530-f146816e158c-kube-api-access-8q4b2\") pod \"cluster-proxy-proxy-agent-7df7df9c64-kzgms\" (UID: \"2a94c6ba-2e5a-4dd5-9530-f146816e158c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"
Sep 14 12:58:36.670410 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.670390    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfc9z\" (UniqueName: \"kubernetes.io/projected/22dbba95-a3d2-4356-a088-55270cfa2f7a-kube-api-access-bfc9z\") pod \"klusterlet-addon-workmgr-795d86c4f-rnppz\" (UID: \"22dbba95-a3d2-4356-a088-55270cfa2f7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795d86c4f-rnppz"
Sep 14 12:58:36.671349 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.671331    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksj79\" (UniqueName: \"kubernetes.io/projected/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-kube-api-access-ksj79\") pod \"dns-default-gw4vc\" (UID: \"f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6\") " pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:58:36.671488 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.671445    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmx2q\" (UniqueName: \"kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-kube-api-access-hmx2q\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:36.671942 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.671919    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nld5n\" (UniqueName: \"kubernetes.io/projected/d44e13a7-7e76-4eee-873f-68bf4066ac29-kube-api-access-nld5n\") pod \"ingress-canary-jmcw8\" (UID: \"d44e13a7-7e76-4eee-873f-68bf4066ac29\") " pod="openshift-ingress-canary/ingress-canary-jmcw8"
Sep 14 12:58:36.702305 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.702268    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b97dp_e8eda6eb-1097-4a02-b60a-45eb7ddcc87e/konnectivity-agent/0.log" limit=1024
Sep 14 12:58:36.730138 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.730115    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"
Sep 14 12:58:36.748694 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.748675    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84458465cd-drbkw"
Sep 14 12:58:36.762897 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.762876    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795d86c4f-rnppz"
Sep 14 12:58:36.768803 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:36.768786    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jmcw8"
Sep 14 12:58:37.164940 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:37.164912    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:37.165289 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:37.164955    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls\") pod \"dns-default-gw4vc\" (UID: \"f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6\") " pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:58:37.165289 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:37.165064    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 12:58:37.165289 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:37.165074    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 12:58:37.165289 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:37.165082    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8565875799-kvq4d: secret "image-registry-tls" not found
Sep 14 12:58:37.165289 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:37.165139    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls podName:f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:38.165123864 +0000 UTC m=+36.599799363 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls") pod "dns-default-gw4vc" (UID: "f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6") : secret "dns-default-metrics-tls" not found
Sep 14 12:58:37.165289 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:37.165156    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls podName:f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e nodeName:}" failed. No retries permitted until 2026-09-14 12:58:38.165150688 +0000 UTC m=+36.599826185 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls") pod "image-registry-8565875799-kvq4d" (UID: "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e") : secret "image-registry-tls" not found
Sep 14 12:58:37.841914 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:37.841632    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-795d86c4f-rnppz"]
Sep 14 12:58:37.846065 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:37.846042    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"]
Sep 14 12:58:37.846965 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:37.846938    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jmcw8"]
Sep 14 12:58:37.847708 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:37.847590    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84458465cd-drbkw"]
Sep 14 12:58:37.850453 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:37.850371    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22dbba95_a3d2_4356_a088_55270cfa2f7a.slice/crio-110c421e6944558280b48549569486b783e7f20bf113eeb8c63b1e78e606dabe WatchSource:0}: Error finding container 110c421e6944558280b48549569486b783e7f20bf113eeb8c63b1e78e606dabe: Status 404 returned error can't find the container with id 110c421e6944558280b48549569486b783e7f20bf113eeb8c63b1e78e606dabe
Sep 14 12:58:37.850635 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:37.850612    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44e13a7_7e76_4eee_873f_68bf4066ac29.slice/crio-57cacd681300e185230ae1ce489a7e90a493b777f38621a3a7eb4804d63de3d6 WatchSource:0}: Error finding container 57cacd681300e185230ae1ce489a7e90a493b777f38621a3a7eb4804d63de3d6: Status 404 returned error can't find the container with id 57cacd681300e185230ae1ce489a7e90a493b777f38621a3a7eb4804d63de3d6
Sep 14 12:58:37.853072 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:37.852499    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podceb40da7_57e5_4547_bd7b_02d0c3c3e8ef.slice/crio-c537e7e76e0aa48f5887afdd4eef87f5763923fffbf0b9075960b39e365ec0e2 WatchSource:0}: Error finding container c537e7e76e0aa48f5887afdd4eef87f5763923fffbf0b9075960b39e365ec0e2: Status 404 returned error can't find the container with id c537e7e76e0aa48f5887afdd4eef87f5763923fffbf0b9075960b39e365ec0e2
Sep 14 12:58:37.853329 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:58:37.853233    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a94c6ba_2e5a_4dd5_9530_f146816e158c.slice/crio-1f83d7235008512c2c347949f06418e1f0045dfb39b765675431c5200bb6851b WatchSource:0}: Error finding container 1f83d7235008512c2c347949f06418e1f0045dfb39b765675431c5200bb6851b: Status 404 returned error can't find the container with id 1f83d7235008512c2c347949f06418e1f0045dfb39b765675431c5200bb6851b
Sep 14 12:58:38.043147 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:38.043120    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:58:38.043147 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:38.043137    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:58:38.043282 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:38.043124    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:58:38.046449 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:38.046430    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 14 12:58:38.046449 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:38.046439    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tlghq\""
Sep 14 12:58:38.046591 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:38.046549    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 14 12:58:38.047513 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:38.047108    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9l5s5\""
Sep 14 12:58:38.047513 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:38.047228    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 14 12:58:38.047513 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:38.047252    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 14 12:58:38.172915 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:38.172895    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:38.173401 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:38.172925    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls\") pod \"dns-default-gw4vc\" (UID: \"f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6\") " pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:58:38.173401 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:38.173031    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 12:58:38.173401 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:38.173084    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls podName:f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:40.17307021 +0000 UTC m=+38.607745706 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls") pod "dns-default-gw4vc" (UID: "f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6") : secret "dns-default-metrics-tls" not found
Sep 14 12:58:38.173401 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:38.173032    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 12:58:38.173401 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:38.173134    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8565875799-kvq4d: secret "image-registry-tls" not found
Sep 14 12:58:38.173401 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:38.173186    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls podName:f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e nodeName:}" failed. No retries permitted until 2026-09-14 12:58:40.173171454 +0000 UTC m=+38.607846971 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls") pod "image-registry-8565875799-kvq4d" (UID: "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e") : secret "image-registry-tls" not found
Sep 14 12:58:38.410090 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:38.410022    2557 generic.go:356] "Generic (PLEG): container finished" podID="f26cce38-f34e-4f48-9a02-88e6674fe266" containerID="91dca534ed3e29d18fca9bc6ed20d35dc00f361f7317b0de0b64961cd6fcd116" exitCode=0
Sep 14 12:58:38.410213 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:38.410117    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcv75" event={"ID":"f26cce38-f34e-4f48-9a02-88e6674fe266","Type":"ContainerDied","Data":"91dca534ed3e29d18fca9bc6ed20d35dc00f361f7317b0de0b64961cd6fcd116"}
Sep 14 12:58:38.411507 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:38.411486    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795d86c4f-rnppz" event={"ID":"22dbba95-a3d2-4356-a088-55270cfa2f7a","Type":"ContainerStarted","Data":"110c421e6944558280b48549569486b783e7f20bf113eeb8c63b1e78e606dabe"}
Sep 14 12:58:38.412879 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:38.412849    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms" event={"ID":"2a94c6ba-2e5a-4dd5-9530-f146816e158c","Type":"ContainerStarted","Data":"1f83d7235008512c2c347949f06418e1f0045dfb39b765675431c5200bb6851b"}
Sep 14 12:58:38.414279 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:38.414260    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jmcw8" event={"ID":"d44e13a7-7e76-4eee-873f-68bf4066ac29","Type":"ContainerStarted","Data":"57cacd681300e185230ae1ce489a7e90a493b777f38621a3a7eb4804d63de3d6"}
Sep 14 12:58:38.415423 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:38.415400    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84458465cd-drbkw" event={"ID":"ceb40da7-57e5-4547-bd7b-02d0c3c3e8ef","Type":"ContainerStarted","Data":"c537e7e76e0aa48f5887afdd4eef87f5763923fffbf0b9075960b39e365ec0e2"}
Sep 14 12:58:39.428575 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:39.428539    2557 generic.go:356] "Generic (PLEG): container finished" podID="f26cce38-f34e-4f48-9a02-88e6674fe266" containerID="32f911cc90014db93d2d099845ec3ff76478f7e77492346d7be989a814cb35f2" exitCode=0
Sep 14 12:58:39.429055 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:39.428608    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcv75" event={"ID":"f26cce38-f34e-4f48-9a02-88e6674fe266","Type":"ContainerDied","Data":"32f911cc90014db93d2d099845ec3ff76478f7e77492346d7be989a814cb35f2"}
Sep 14 12:58:40.125077 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:40.125023    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b97dp_e8eda6eb-1097-4a02-b60a-45eb7ddcc87e/konnectivity-agent/0.log" limit=1024
Sep 14 12:58:40.188211 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:40.188113    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b97dp_e8eda6eb-1097-4a02-b60a-45eb7ddcc87e/konnectivity-agent/0.log" limit=1024
Sep 14 12:58:40.192355 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:40.191576    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:40.192355 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:40.191641    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls\") pod \"dns-default-gw4vc\" (UID: \"f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6\") " pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:58:40.192355 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:40.191835    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 12:58:40.192355 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:40.191892    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls podName:f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:44.191875033 +0000 UTC m=+42.626550533 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls") pod "dns-default-gw4vc" (UID: "f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6") : secret "dns-default-metrics-tls" not found
Sep 14 12:58:40.192355 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:40.192267    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 12:58:40.192355 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:40.192291    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8565875799-kvq4d: secret "image-registry-tls" not found
Sep 14 12:58:40.192355 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:40.192334    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls podName:f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e nodeName:}" failed. No retries permitted until 2026-09-14 12:58:44.192319805 +0000 UTC m=+42.626995309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls") pod "image-registry-8565875799-kvq4d" (UID: "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e") : secret "image-registry-tls" not found
Sep 14 12:58:40.437456 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:40.437421    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcv75" event={"ID":"f26cce38-f34e-4f48-9a02-88e6674fe266","Type":"ContainerStarted","Data":"f39baf2c3b6eeca18ae0f02cc2a104c1ee193ea1dad7a51403b348eceb364023"}
Sep 14 12:58:40.458951 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:40.458872    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b97dp_e8eda6eb-1097-4a02-b60a-45eb7ddcc87e/konnectivity-agent/0.log" limit=1024
Sep 14 12:58:40.463005 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:40.462955    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zcv75" podStartSLOduration=8.050279158 podStartE2EDuration="38.462932484s" totalImagesPullingTime="30.412653326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:58:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:40.460851932 +0000 UTC m=+38.895527452" watchObservedRunningTime="2026-09-14 12:58:40.462932484 +0000 UTC m=+38.897608003"
Sep 14 12:58:40.559351 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:40.559306    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b97dp_e8eda6eb-1097-4a02-b60a-45eb7ddcc87e/konnectivity-agent/0.log" limit=1024
Sep 14 12:58:40.660624 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:40.659716    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b97dp_e8eda6eb-1097-4a02-b60a-45eb7ddcc87e/konnectivity-agent/0.log" limit=1024
Sep 14 12:58:40.723223 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:40.723148    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b97dp_e8eda6eb-1097-4a02-b60a-45eb7ddcc87e/konnectivity-agent/0.log" limit=1024
Sep 14 12:58:42.092194 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:42.092072    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b97dp_e8eda6eb-1097-4a02-b60a-45eb7ddcc87e/konnectivity-agent/0.log" limit=1024
Sep 14 12:58:42.152243 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:42.152206    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b97dp_e8eda6eb-1097-4a02-b60a-45eb7ddcc87e/konnectivity-agent/0.log" limit=1024
Sep 14 12:58:43.077296 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:43.077256    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b97dp_e8eda6eb-1097-4a02-b60a-45eb7ddcc87e/konnectivity-agent/0.log" limit=1024
Sep 14 12:58:43.130861 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:43.130821    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b97dp_e8eda6eb-1097-4a02-b60a-45eb7ddcc87e/konnectivity-agent/0.log" limit=1024
Sep 14 12:58:44.221803 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:44.221766    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:44.222214 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:44.221823    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls\") pod \"dns-default-gw4vc\" (UID: \"f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6\") " pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:58:44.222214 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:44.221923    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 12:58:44.222214 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:44.221998    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls podName:f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:52.221983877 +0000 UTC m=+50.656659379 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls") pod "dns-default-gw4vc" (UID: "f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6") : secret "dns-default-metrics-tls" not found
Sep 14 12:58:44.222214 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:44.221926    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 12:58:44.222214 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:44.222027    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8565875799-kvq4d: secret "image-registry-tls" not found
Sep 14 12:58:44.222214 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:44.222067    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls podName:f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e nodeName:}" failed. No retries permitted until 2026-09-14 12:58:52.222056756 +0000 UTC m=+50.656732266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls") pod "image-registry-8565875799-kvq4d" (UID: "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e") : secret "image-registry-tls" not found
Sep 14 12:58:47.102985 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:47.102921    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b97dp_e8eda6eb-1097-4a02-b60a-45eb7ddcc87e/konnectivity-agent/0.log" limit=1024
Sep 14 12:58:47.459409 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:47.459373    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jmcw8" event={"ID":"d44e13a7-7e76-4eee-873f-68bf4066ac29","Type":"ContainerStarted","Data":"67dc8b7336d04364bcaadffcfb879ae709aa6f349c655f22fdbe2b2a361083c5"}
Sep 14 12:58:47.460913 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:47.460886    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84458465cd-drbkw" event={"ID":"ceb40da7-57e5-4547-bd7b-02d0c3c3e8ef","Type":"ContainerStarted","Data":"e51d4944033cf4476e5c17452f3c0197b34353250588efbff29606fc938dd397"}
Sep 14 12:58:47.462334 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:47.462307    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795d86c4f-rnppz" event={"ID":"22dbba95-a3d2-4356-a088-55270cfa2f7a","Type":"ContainerStarted","Data":"576f4ae1aaaf467230d62f63d0d5ce55634c6b8ef99ffd42ec117a85c71e15b5"}
Sep 14 12:58:47.462438 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:47.462357    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795d86c4f-rnppz"
Sep 14 12:58:47.464358 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:47.464333    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795d86c4f-rnppz"
Sep 14 12:58:47.464622 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:47.464603    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms" event={"ID":"2a94c6ba-2e5a-4dd5-9530-f146816e158c","Type":"ContainerStarted","Data":"c11ffbf297c8321eb8f403b7e72211c6c3e40ad0e902e144fa2af71a0ca3891a"}
Sep 14 12:58:47.464622 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:47.464631    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms" event={"ID":"2a94c6ba-2e5a-4dd5-9530-f146816e158c","Type":"ContainerStarted","Data":"82050dd432d99f6b00717458992abaadebd05b2988a3172a10825b6b7564423e"}
Sep 14 12:58:47.464793 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:47.464646    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms" event={"ID":"2a94c6ba-2e5a-4dd5-9530-f146816e158c","Type":"ContainerStarted","Data":"321ce66699a526394687b1eb77afba72228c015777ac7c1afba046031ba0f561"}
Sep 14 12:58:47.481991 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:47.481949    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jmcw8" podStartSLOduration=2.747693528 podStartE2EDuration="11.481935999s" totalImagesPullingTime="8.734242471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:47.480450447 +0000 UTC m=+45.915125966" watchObservedRunningTime="2026-09-14 12:58:47.481935999 +0000 UTC m=+45.916611512"
Sep 14 12:58:47.502435 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:47.502399    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795d86c4f-rnppz" podStartSLOduration=22.727551878 podStartE2EDuration="31.50238611s" totalImagesPullingTime="8.774834232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:47.501642678 +0000 UTC m=+45.936318197" watchObservedRunningTime="2026-09-14 12:58:47.50238611 +0000 UTC m=+45.937061622"
Sep 14 12:58:47.521766 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:47.521717    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84458465cd-drbkw" podStartSLOduration=22.787842227 podStartE2EDuration="31.521705254s" totalImagesPullingTime="8.733863027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:47.521376056 +0000 UTC m=+45.956051576" watchObservedRunningTime="2026-09-14 12:58:47.521705254 +0000 UTC m=+45.956380773"
Sep 14 12:58:47.543338 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:47.543304    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms" podStartSLOduration=22.786121268 podStartE2EDuration="31.543293804s" totalImagesPullingTime="8.757172536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:47.543095808 +0000 UTC m=+45.977771328" watchObservedRunningTime="2026-09-14 12:58:47.543293804 +0000 UTC m=+45.977969323"
Sep 14 12:58:52.278587 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:52.278551    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:58:52.278587 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:58:52.278592    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls\") pod \"dns-default-gw4vc\" (UID: \"f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6\") " pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:58:52.279008 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:52.278678    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 12:58:52.279008 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:52.278684    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 12:58:52.279008 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:52.278701    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8565875799-kvq4d: secret "image-registry-tls" not found
Sep 14 12:58:52.279008 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:52.278726    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls podName:f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:08.278712303 +0000 UTC m=+66.713387814 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls") pod "dns-default-gw4vc" (UID: "f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6") : secret "dns-default-metrics-tls" not found
Sep 14 12:58:52.279008 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:58:52.278764    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls podName:f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:08.278731935 +0000 UTC m=+66.713407437 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls") pod "image-registry-8565875799-kvq4d" (UID: "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e") : secret "image-registry-tls" not found
Sep 14 12:59:02.401533 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:02.401508    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jwqnf"
Sep 14 12:59:06.782609 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:06.782576    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs\") pod \"network-metrics-daemon-nl2vh\" (UID: \"898a4541-4e3f-4276-b037-ff179ac5fc52\") " pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 12:59:06.785791 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:06.785771    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 14 12:59:06.793343 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:06.793321    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 14 12:59:06.793440 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:06.793396    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs podName:898a4541-4e3f-4276-b037-ff179ac5fc52 nodeName:}" failed. No retries permitted until 2026-09-14 13:00:10.793381726 +0000 UTC m=+129.228057222 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs") pod "network-metrics-daemon-nl2vh" (UID: "898a4541-4e3f-4276-b037-ff179ac5fc52") : secret "metrics-daemon-secret" not found
Sep 14 12:59:06.883508 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:06.883484    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret\") pod \"global-pull-secret-syncer-zgj9g\" (UID: \"7eb7a234-f263-457d-9f06-c895898890ac\") " pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:59:06.886330 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:06.886313    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 14 12:59:06.896856 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:06.896830    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7eb7a234-f263-457d-9f06-c895898890ac-original-pull-secret\") pod \"global-pull-secret-syncer-zgj9g\" (UID: \"7eb7a234-f263-457d-9f06-c895898890ac\") " pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:59:06.983852 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:06.983833    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbtl\" (UniqueName: \"kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl\") pod \"network-check-target-jcjld\" (UID: \"2fafcfb0-af2a-41f5-8742-e9679b873e89\") " pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:59:06.986853 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:06.986839    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 14 12:59:06.996990 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:06.996976    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 14 12:59:07.006428 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.006407    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lbtl\" (UniqueName: \"kubernetes.io/projected/2fafcfb0-af2a-41f5-8742-e9679b873e89-kube-api-access-7lbtl\") pod \"network-check-target-jcjld\" (UID: \"2fafcfb0-af2a-41f5-8742-e9679b873e89\") " pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:59:07.164278 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.164223    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tlghq\""
Sep 14 12:59:07.166339 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.166324    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgj9g"
Sep 14 12:59:07.172352 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.172334    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:59:07.435652 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.435581    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zgj9g"]
Sep 14 12:59:07.438530 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:59:07.438499    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb7a234_f263_457d_9f06_c895898890ac.slice/crio-c92599c451fefd596fc3d75011190c9c934e3a38edd884180ea4736f821f055c WatchSource:0}: Error finding container c92599c451fefd596fc3d75011190c9c934e3a38edd884180ea4736f821f055c: Status 404 returned error can't find the container with id c92599c451fefd596fc3d75011190c9c934e3a38edd884180ea4736f821f055c
Sep 14 12:59:07.442157 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.442137    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jcjld"]
Sep 14 12:59:07.445054 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:59:07.445031    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fafcfb0_af2a_41f5_8742_e9679b873e89.slice/crio-e4f736e913f50b2052252b58a3d7a275b1df333fdb0179e6b08d05e94f6c11b9 WatchSource:0}: Error finding container e4f736e913f50b2052252b58a3d7a275b1df333fdb0179e6b08d05e94f6c11b9: Status 404 returned error can't find the container with id e4f736e913f50b2052252b58a3d7a275b1df333fdb0179e6b08d05e94f6c11b9
Sep 14 12:59:07.525631 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.525603    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jcjld" event={"ID":"2fafcfb0-af2a-41f5-8742-e9679b873e89","Type":"ContainerStarted","Data":"e4f736e913f50b2052252b58a3d7a275b1df333fdb0179e6b08d05e94f6c11b9"}
Sep 14 12:59:07.526757 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.526722    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zgj9g" event={"ID":"7eb7a234-f263-457d-9f06-c895898890ac","Type":"ContainerStarted","Data":"c92599c451fefd596fc3d75011190c9c934e3a38edd884180ea4736f821f055c"}
Sep 14 12:59:07.628018 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.627989    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-97j4b"]
Sep 14 12:59:07.631343 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.631322    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:07.634929 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.634907    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 14 12:59:07.634929 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.634922    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 14 12:59:07.635067 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.634922    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 14 12:59:07.636328 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.636303    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 14 12:59:07.636459 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.636307    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-26vf9\""
Sep 14 12:59:07.641441 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.641420    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-97j4b"]
Sep 14 12:59:07.689766 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.689732    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:07.689879 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.689779    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-data-volume\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:07.689879 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.689796    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w284s\" (UniqueName: \"kubernetes.io/projected/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-kube-api-access-w284s\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:07.689879 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.689855    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:07.690031 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.689901    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-crio-socket\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:07.791176 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.791149    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-data-volume\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:07.791563 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.791180    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w284s\" (UniqueName: \"kubernetes.io/projected/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-kube-api-access-w284s\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:07.791563 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.791204    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:07.791563 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.791221    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-crio-socket\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:07.791563 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.791296    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:07.791563 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:07.791312    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 12:59:07.791563 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:07.791404    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 12:59:07.791563 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:07.791421    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-kube-rbac-proxy-cm podName:5825fb8c-8360-46b3-9f34-57ab3a5a3e4e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:08.291396833 +0000 UTC m=+66.726072333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-97j4b" (UID: "5825fb8c-8360-46b3-9f34-57ab3a5a3e4e") : configmap "kube-rbac-proxy" not found
Sep 14 12:59:07.791563 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:07.791450    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-insights-runtime-extractor-tls podName:5825fb8c-8360-46b3-9f34-57ab3a5a3e4e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:08.291437044 +0000 UTC m=+66.726112541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-97j4b" (UID: "5825fb8c-8360-46b3-9f34-57ab3a5a3e4e") : secret "insights-runtime-extractor-tls" not found
Sep 14 12:59:07.791563 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.791403    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-crio-socket\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:07.791925 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.791600    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-data-volume\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:07.802788 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:07.802764    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w284s\" (UniqueName: \"kubernetes.io/projected/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-kube-api-access-w284s\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:08.295876 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:08.295838    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:08.296048 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:08.295893    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:08.296048 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:08.295921    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:59:08.296048 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:08.295946    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls\") pod \"dns-default-gw4vc\" (UID: \"f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6\") " pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:59:08.296048 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:08.295996    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 12:59:08.296048 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:08.296037    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 12:59:08.296329 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:08.296072    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-insights-runtime-extractor-tls podName:5825fb8c-8360-46b3-9f34-57ab3a5a3e4e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:09.296051615 +0000 UTC m=+67.730727129 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-97j4b" (UID: "5825fb8c-8360-46b3-9f34-57ab3a5a3e4e") : secret "insights-runtime-extractor-tls" not found
Sep 14 12:59:08.296329 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:08.296079    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 12:59:08.296329 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:08.296097    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8565875799-kvq4d: secret "image-registry-tls" not found
Sep 14 12:59:08.296329 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:08.296104    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-kube-rbac-proxy-cm podName:5825fb8c-8360-46b3-9f34-57ab3a5a3e4e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:09.296086723 +0000 UTC m=+67.730762248 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-97j4b" (UID: "5825fb8c-8360-46b3-9f34-57ab3a5a3e4e") : configmap "kube-rbac-proxy" not found
Sep 14 12:59:08.296329 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:08.296141    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 12:59:08.296329 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:08.296144    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls podName:f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:40.29612737 +0000 UTC m=+98.730802883 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls") pod "image-registry-8565875799-kvq4d" (UID: "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e") : secret "image-registry-tls" not found
Sep 14 12:59:08.296329 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:08.296191    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls podName:f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:40.296179539 +0000 UTC m=+98.730855038 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls") pod "dns-default-gw4vc" (UID: "f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6") : secret "dns-default-metrics-tls" not found
Sep 14 12:59:09.306699 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:09.303619    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:09.306699 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:09.303795    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 12:59:09.306699 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:09.303877    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-kube-rbac-proxy-cm podName:5825fb8c-8360-46b3-9f34-57ab3a5a3e4e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:11.303858967 +0000 UTC m=+69.738534485 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-97j4b" (UID: "5825fb8c-8360-46b3-9f34-57ab3a5a3e4e") : configmap "kube-rbac-proxy" not found
Sep 14 12:59:09.306699 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:09.304774    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:09.306699 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:09.305027    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 12:59:09.306699 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:09.305106    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-insights-runtime-extractor-tls podName:5825fb8c-8360-46b3-9f34-57ab3a5a3e4e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:11.305089056 +0000 UTC m=+69.739764568 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-97j4b" (UID: "5825fb8c-8360-46b3-9f34-57ab3a5a3e4e") : secret "insights-runtime-extractor-tls" not found
Sep 14 12:59:10.397103 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:10.397076    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nhvd2_e41b38b6-a3f2-4093-977d-c44ada18fdcb/dns-node-resolver/0.log"
Sep 14 12:59:11.320626 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:11.320592    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:11.320813 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:11.320702    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:11.320813 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:11.320730    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 12:59:11.320934 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:11.320813    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 12:59:11.320934 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:11.320831    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-kube-rbac-proxy-cm podName:5825fb8c-8360-46b3-9f34-57ab3a5a3e4e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:15.320813545 +0000 UTC m=+73.755489046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-97j4b" (UID: "5825fb8c-8360-46b3-9f34-57ab3a5a3e4e") : configmap "kube-rbac-proxy" not found
Sep 14 12:59:11.320934 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:11.320870    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-insights-runtime-extractor-tls podName:5825fb8c-8360-46b3-9f34-57ab3a5a3e4e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:15.320857569 +0000 UTC m=+73.755533066 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-97j4b" (UID: "5825fb8c-8360-46b3-9f34-57ab3a5a3e4e") : secret "insights-runtime-extractor-tls" not found
Sep 14 12:59:11.397990 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:11.397962    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5vcr5_b4458034-12ab-40d0-93c8-77de00c6ffb7/node-ca/0.log"
Sep 14 12:59:12.196859 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:12.196831    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jmcw8_d44e13a7-7e76-4eee-873f-68bf4066ac29/serve-healthcheck-canary/0.log"
Sep 14 12:59:12.547053 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:12.547022    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jcjld" event={"ID":"2fafcfb0-af2a-41f5-8742-e9679b873e89","Type":"ContainerStarted","Data":"206c549da7d5b8a9a13c64eb7f5404ac8024ca26bf9b4ea2badf018eaff2931e"}
Sep 14 12:59:12.547433 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:12.547123    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:59:12.548786 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:12.548764    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zgj9g" event={"ID":"7eb7a234-f263-457d-9f06-c895898890ac","Type":"ContainerStarted","Data":"0628bdf9545fd15c6e8184120d2c8684b6805372d53a44eea76f16eb11514a2c"}
Sep 14 12:59:12.564924 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:12.564883    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jcjld" podStartSLOduration=65.031470394 podStartE2EDuration="1m9.564869991s" totalImagesPullingTime="4.533399597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:12.563852954 +0000 UTC m=+70.998528475" watchObservedRunningTime="2026-09-14 12:59:12.564869991 +0000 UTC m=+70.999545502"
Sep 14 12:59:12.579031 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:12.578993    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zgj9g" podStartSLOduration=65.035752871 podStartE2EDuration="1m9.578981015s" totalImagesPullingTime="4.543228144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:12.57870262 +0000 UTC m=+71.013378136" watchObservedRunningTime="2026-09-14 12:59:12.578981015 +0000 UTC m=+71.013656533"
Sep 14 12:59:15.349734 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:15.349700    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:15.350281 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:15.349788    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:15.350281 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:15.349827    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 12:59:15.350281 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:15.349878    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 12:59:15.350281 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:15.349909    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-kube-rbac-proxy-cm podName:5825fb8c-8360-46b3-9f34-57ab3a5a3e4e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:23.349886564 +0000 UTC m=+81.784562083 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-97j4b" (UID: "5825fb8c-8360-46b3-9f34-57ab3a5a3e4e") : configmap "kube-rbac-proxy" not found
Sep 14 12:59:15.350281 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:15.349929    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-insights-runtime-extractor-tls podName:5825fb8c-8360-46b3-9f34-57ab3a5a3e4e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:23.349920175 +0000 UTC m=+81.784595677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-97j4b" (UID: "5825fb8c-8360-46b3-9f34-57ab3a5a3e4e") : secret "insights-runtime-extractor-tls" not found
Sep 14 12:59:23.408949 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:23.408905    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:23.408949 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:23.408955    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:23.409422 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:23.409048    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 12:59:23.409422 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:23.409111    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-insights-runtime-extractor-tls podName:5825fb8c-8360-46b3-9f34-57ab3a5a3e4e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:39.409093753 +0000 UTC m=+97.843769254 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-97j4b" (UID: "5825fb8c-8360-46b3-9f34-57ab3a5a3e4e") : secret "insights-runtime-extractor-tls" not found
Sep 14 12:59:23.409499 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:23.409461    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:39.420299 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:39.420174    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:39.422474 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:39.422447    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5825fb8c-8360-46b3-9f34-57ab3a5a3e4e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-97j4b\" (UID: \"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e\") " pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:39.441297 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:39.441275    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-97j4b"
Sep 14 12:59:39.710798 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:39.710768    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-97j4b"]
Sep 14 12:59:39.713440 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:59:39.713414    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5825fb8c_8360_46b3_9f34_57ab3a5a3e4e.slice/crio-2988d6b5b48020416eacabec42c88d900bd414d8bddc9c3f81679f5aaec6e154 WatchSource:0}: Error finding container 2988d6b5b48020416eacabec42c88d900bd414d8bddc9c3f81679f5aaec6e154: Status 404 returned error can't find the container with id 2988d6b5b48020416eacabec42c88d900bd414d8bddc9c3f81679f5aaec6e154
Sep 14 12:59:40.327672 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:40.327640    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:59:40.327848 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:40.327679    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls\") pod \"dns-default-gw4vc\" (UID: \"f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6\") " pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:59:40.329854 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:40.329834    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6-metrics-tls\") pod \"dns-default-gw4vc\" (UID: \"f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6\") " pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:59:40.329970 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:40.329876    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls\") pod \"image-registry-8565875799-kvq4d\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") " pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:59:40.344370 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:40.344342    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8dx6q\""
Sep 14 12:59:40.352713 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:40.352698    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:59:40.378208 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:40.378105    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xjstd\""
Sep 14 12:59:40.386488 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:40.386460    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:59:40.633293 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:40.633225    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-97j4b" event={"ID":"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e","Type":"ContainerStarted","Data":"454fbc65e87e289eb2d9b550f8975c7e55dd0c81faa2cecda81b0e9d731508bb"}
Sep 14 12:59:40.633293 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:40.633281    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-97j4b" event={"ID":"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e","Type":"ContainerStarted","Data":"6d93e94a7de5bd8849cd8aff16e3b71162b59d6ea9e26ba013cad2a0fc130045"}
Sep 14 12:59:40.633628 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:40.633300    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-97j4b" event={"ID":"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e","Type":"ContainerStarted","Data":"2988d6b5b48020416eacabec42c88d900bd414d8bddc9c3f81679f5aaec6e154"}
Sep 14 12:59:40.643601 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:40.643576    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8565875799-kvq4d"]
Sep 14 12:59:40.646675 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:59:40.646650    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bf9a24_6ac3_402d_b91a_f0fed1f13a3e.slice/crio-7ab1ad50599bd53213f0a091864fc36a3fcbe524f586b7eccee86ba3c293d8d7 WatchSource:0}: Error finding container 7ab1ad50599bd53213f0a091864fc36a3fcbe524f586b7eccee86ba3c293d8d7: Status 404 returned error can't find the container with id 7ab1ad50599bd53213f0a091864fc36a3fcbe524f586b7eccee86ba3c293d8d7
Sep 14 12:59:40.658690 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:40.658669    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gw4vc"]
Sep 14 12:59:40.661550 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:59:40.661526    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d17f3f_bf92_4d1f_a6b8_b1efdd0781e6.slice/crio-056fee14c6d82550f2e98abcb05762375d7531d543d02eac1685080d9d325ead WatchSource:0}: Error finding container 056fee14c6d82550f2e98abcb05762375d7531d543d02eac1685080d9d325ead: Status 404 returned error can't find the container with id 056fee14c6d82550f2e98abcb05762375d7531d543d02eac1685080d9d325ead
Sep 14 12:59:41.638596 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:41.638558    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8565875799-kvq4d" event={"ID":"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e","Type":"ContainerStarted","Data":"b5fc440de29b1ca4801675fdf182946af310d90e3ef478b2770308bbec07e942"}
Sep 14 12:59:41.639062 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:41.638618    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8565875799-kvq4d" event={"ID":"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e","Type":"ContainerStarted","Data":"7ab1ad50599bd53213f0a091864fc36a3fcbe524f586b7eccee86ba3c293d8d7"}
Sep 14 12:59:41.639062 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:41.638649    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:59:41.639985 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:41.639959    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gw4vc" event={"ID":"f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6","Type":"ContainerStarted","Data":"056fee14c6d82550f2e98abcb05762375d7531d543d02eac1685080d9d325ead"}
Sep 14 12:59:41.660792 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:41.660725    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8565875799-kvq4d" podStartSLOduration=98.660707157 podStartE2EDuration="1m38.660707157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:41.658789817 +0000 UTC m=+100.093465343" watchObservedRunningTime="2026-09-14 12:59:41.660707157 +0000 UTC m=+100.095382677"
Sep 14 12:59:42.645290 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:42.645212    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-97j4b" event={"ID":"5825fb8c-8360-46b3-9f34-57ab3a5a3e4e","Type":"ContainerStarted","Data":"ea660ccd0fecda25f8abbf2103681b11d9955bd00c9a867cb06e0d5ef9e5fd64"}
Sep 14 12:59:42.647180 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:42.647153    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gw4vc" event={"ID":"f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6","Type":"ContainerStarted","Data":"62127bc5f801ccc1c23a55bf28fe3589e65b2b656d27c7d8a84ad151503cdd99"}
Sep 14 12:59:42.647298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:42.647212    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:59:42.647298 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:42.647238    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gw4vc" event={"ID":"f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6","Type":"ContainerStarted","Data":"f68d964ea0fbec910c3228a032d388ac67cb01ab1be060f2383f53303ddbb3f2"}
Sep 14 12:59:42.665289 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:42.665241    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-97j4b" podStartSLOduration=33.296274357 podStartE2EDuration="35.665226088s" totalImagesPullingTime="2.368951731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:59:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:42.663484612 +0000 UTC m=+101.098160130" watchObservedRunningTime="2026-09-14 12:59:42.665226088 +0000 UTC m=+101.099901606"
Sep 14 12:59:42.682562 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:42.682519    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gw4vc" podStartSLOduration=65.134521353 podStartE2EDuration="1m6.682506259s" totalImagesPullingTime="1.547984906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:42.681388134 +0000 UTC m=+101.116063654" watchObservedRunningTime="2026-09-14 12:59:42.682506259 +0000 UTC m=+101.117181778"
Sep 14 12:59:43.554657 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:43.554625    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jcjld"
Sep 14 12:59:52.652034 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:52.652002    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gw4vc"
Sep 14 12:59:54.169877 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.169848    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bcclt"]
Sep 14 12:59:54.172800 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.172783    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.175329 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.175308    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 14 12:59:54.175729 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.175712    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 14 12:59:54.175796 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.175733    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5rv2c\""
Sep 14 12:59:54.177115 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.177094    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 14 12:59:54.177176 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.177105    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 14 12:59:54.177176 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.177157    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 14 12:59:54.177352 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.177332    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 14 12:59:54.318249 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.318226    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/84e3b9c9-763e-405c-b22f-e5c2b07156d3-root\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.318369 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.318266    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/84e3b9c9-763e-405c-b22f-e5c2b07156d3-node-exporter-textfile\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.318369 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.318295    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84e3b9c9-763e-405c-b22f-e5c2b07156d3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.318369 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.318350    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptgpp\" (UniqueName: \"kubernetes.io/projected/84e3b9c9-763e-405c-b22f-e5c2b07156d3-kube-api-access-ptgpp\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.318466 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.318409    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84e3b9c9-763e-405c-b22f-e5c2b07156d3-metrics-client-ca\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.318466 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.318431    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/84e3b9c9-763e-405c-b22f-e5c2b07156d3-node-exporter-wtmp\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.318533 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.318473    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84e3b9c9-763e-405c-b22f-e5c2b07156d3-node-exporter-tls\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.318533 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.318503    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84e3b9c9-763e-405c-b22f-e5c2b07156d3-sys\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.318533 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.318525    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/84e3b9c9-763e-405c-b22f-e5c2b07156d3-node-exporter-accelerators-collector-config\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.419693 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.419658    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptgpp\" (UniqueName: \"kubernetes.io/projected/84e3b9c9-763e-405c-b22f-e5c2b07156d3-kube-api-access-ptgpp\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.419856 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.419715    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84e3b9c9-763e-405c-b22f-e5c2b07156d3-metrics-client-ca\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.419856 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.419843    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/84e3b9c9-763e-405c-b22f-e5c2b07156d3-node-exporter-wtmp\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.420009 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.419875    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84e3b9c9-763e-405c-b22f-e5c2b07156d3-node-exporter-tls\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.420009 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.419897    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84e3b9c9-763e-405c-b22f-e5c2b07156d3-sys\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.420009 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.419919    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/84e3b9c9-763e-405c-b22f-e5c2b07156d3-node-exporter-accelerators-collector-config\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.420009 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.419936    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/84e3b9c9-763e-405c-b22f-e5c2b07156d3-root\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.420009 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.419962    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/84e3b9c9-763e-405c-b22f-e5c2b07156d3-node-exporter-textfile\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.420009 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.420006    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84e3b9c9-763e-405c-b22f-e5c2b07156d3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.420307 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.420018    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84e3b9c9-763e-405c-b22f-e5c2b07156d3-sys\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.420307 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.420025    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/84e3b9c9-763e-405c-b22f-e5c2b07156d3-root\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.420307 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.420005    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/84e3b9c9-763e-405c-b22f-e5c2b07156d3-node-exporter-wtmp\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.420307 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:54.420144    2557 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 14 12:59:54.420307 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:54.420222    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84e3b9c9-763e-405c-b22f-e5c2b07156d3-node-exporter-tls podName:84e3b9c9-763e-405c-b22f-e5c2b07156d3 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:54.920200709 +0000 UTC m=+113.354876208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/84e3b9c9-763e-405c-b22f-e5c2b07156d3-node-exporter-tls") pod "node-exporter-bcclt" (UID: "84e3b9c9-763e-405c-b22f-e5c2b07156d3") : secret "node-exporter-tls" not found
Sep 14 12:59:54.420307 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.420250    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/84e3b9c9-763e-405c-b22f-e5c2b07156d3-node-exporter-textfile\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.420529 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.420336    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84e3b9c9-763e-405c-b22f-e5c2b07156d3-metrics-client-ca\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.420529 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.420427    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/84e3b9c9-763e-405c-b22f-e5c2b07156d3-node-exporter-accelerators-collector-config\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.422156 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.422137    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84e3b9c9-763e-405c-b22f-e5c2b07156d3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.428062 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.428040    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptgpp\" (UniqueName: \"kubernetes.io/projected/84e3b9c9-763e-405c-b22f-e5c2b07156d3-kube-api-access-ptgpp\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.924327 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:54.924291    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84e3b9c9-763e-405c-b22f-e5c2b07156d3-node-exporter-tls\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:54.924487 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:54.924440    2557 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 14 12:59:54.924530 ip-10-0-129-61 kubenswrapper[2557]: E0914 12:59:54.924504    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84e3b9c9-763e-405c-b22f-e5c2b07156d3-node-exporter-tls podName:84e3b9c9-763e-405c-b22f-e5c2b07156d3 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:55.924490033 +0000 UTC m=+114.359165534 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/84e3b9c9-763e-405c-b22f-e5c2b07156d3-node-exporter-tls") pod "node-exporter-bcclt" (UID: "84e3b9c9-763e-405c-b22f-e5c2b07156d3") : secret "node-exporter-tls" not found
Sep 14 12:59:55.743479 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:55.743442    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8565875799-kvq4d"]
Sep 14 12:59:55.748046 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:55.748023    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 12:59:55.932588 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:55.932538    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84e3b9c9-763e-405c-b22f-e5c2b07156d3-node-exporter-tls\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:55.934815 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:55.934788    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84e3b9c9-763e-405c-b22f-e5c2b07156d3-node-exporter-tls\") pod \"node-exporter-bcclt\" (UID: \"84e3b9c9-763e-405c-b22f-e5c2b07156d3\") " pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:55.986035 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:55.986008    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bcclt"
Sep 14 12:59:55.995950 ip-10-0-129-61 kubenswrapper[2557]: W0914 12:59:55.995895    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e3b9c9_763e_405c_b22f_e5c2b07156d3.slice/crio-22573a462607df2c6097dd03fe194bf4cc0b9c98e27699180765987ddcf8d41b WatchSource:0}: Error finding container 22573a462607df2c6097dd03fe194bf4cc0b9c98e27699180765987ddcf8d41b: Status 404 returned error can't find the container with id 22573a462607df2c6097dd03fe194bf4cc0b9c98e27699180765987ddcf8d41b
Sep 14 12:59:56.694189 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:56.694155    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bcclt" event={"ID":"84e3b9c9-763e-405c-b22f-e5c2b07156d3","Type":"ContainerStarted","Data":"22573a462607df2c6097dd03fe194bf4cc0b9c98e27699180765987ddcf8d41b"}
Sep 14 12:59:57.698687 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:57.698653    2557 generic.go:356] "Generic (PLEG): container finished" podID="84e3b9c9-763e-405c-b22f-e5c2b07156d3" containerID="8bc6eaacb9bfd5b79ee7a4b3ed958f246f55bee981f0e5a2e9368d890db1f988" exitCode=0
Sep 14 12:59:57.699188 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:57.698710    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bcclt" event={"ID":"84e3b9c9-763e-405c-b22f-e5c2b07156d3","Type":"ContainerDied","Data":"8bc6eaacb9bfd5b79ee7a4b3ed958f246f55bee981f0e5a2e9368d890db1f988"}
Sep 14 12:59:58.704076 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:58.704036    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bcclt" event={"ID":"84e3b9c9-763e-405c-b22f-e5c2b07156d3","Type":"ContainerStarted","Data":"efb86e56a1e2d0e7b6e3f77ed9b7146e438b67b3101b1b4a8435ac18740f3cf3"}
Sep 14 12:59:58.704452 ip-10-0-129-61 kubenswrapper[2557]: I0914 12:59:58.704091    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bcclt" event={"ID":"84e3b9c9-763e-405c-b22f-e5c2b07156d3","Type":"ContainerStarted","Data":"7d2024b5c2636ff1974356e000866e742be5ad1c0e3a26c5637765dbc8bb7b5a"}
Sep 14 13:00:10.839731 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:10.839692    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs\") pod \"network-metrics-daemon-nl2vh\" (UID: \"898a4541-4e3f-4276-b037-ff179ac5fc52\") " pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 13:00:10.841977 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:10.841953    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898a4541-4e3f-4276-b037-ff179ac5fc52-metrics-certs\") pod \"network-metrics-daemon-nl2vh\" (UID: \"898a4541-4e3f-4276-b037-ff179ac5fc52\") " pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 13:00:11.058264 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:11.058232    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9l5s5\""
Sep 14 13:00:11.066413 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:11.066392    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nl2vh"
Sep 14 13:00:11.338344 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:11.338292    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bcclt" podStartSLOduration=16.426194372 podStartE2EDuration="17.338275693s" totalImagesPullingTime="851.14877ms" totalInitContainerRuntime="60.932551ms" isStatefulPod=true podCreationTimestamp="2026-09-14 12:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:58.725086665 +0000 UTC m=+117.159762185" watchObservedRunningTime="2026-09-14 13:00:11.338275693 +0000 UTC m=+129.772951256"
Sep 14 13:00:11.339081 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:11.339059    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nl2vh"]
Sep 14 13:00:11.341500 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:00:11.341467    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898a4541_4e3f_4276_b037_ff179ac5fc52.slice/crio-f620c79819b90f7d01cdaa4170ee288911617b528cf896dec5365e5c6975236d WatchSource:0}: Error finding container f620c79819b90f7d01cdaa4170ee288911617b528cf896dec5365e5c6975236d: Status 404 returned error can't find the container with id f620c79819b90f7d01cdaa4170ee288911617b528cf896dec5365e5c6975236d
Sep 14 13:00:11.749488 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:11.749454    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nl2vh" event={"ID":"898a4541-4e3f-4276-b037-ff179ac5fc52","Type":"ContainerStarted","Data":"f620c79819b90f7d01cdaa4170ee288911617b528cf896dec5365e5c6975236d"}
Sep 14 13:00:12.754066 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:12.754029    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nl2vh" event={"ID":"898a4541-4e3f-4276-b037-ff179ac5fc52","Type":"ContainerStarted","Data":"6b14873070e6f7d51cfa9cb7cfcf7aef635e16da291d8c29b48832db685f412e"}
Sep 14 13:00:12.754511 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:12.754082    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nl2vh" event={"ID":"898a4541-4e3f-4276-b037-ff179ac5fc52","Type":"ContainerStarted","Data":"3db39b73e2c6a92908086607ce171b98268fa4fdb898ddcdb67deaf90e8c2ce1"}
Sep 14 13:00:12.772565 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:12.772514    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nl2vh" podStartSLOduration=129.815621254 podStartE2EDuration="2m10.772499807s" totalImagesPullingTime="956.878553ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:00:12.77159279 +0000 UTC m=+131.206268309" watchObservedRunningTime="2026-09-14 13:00:12.772499807 +0000 UTC m=+131.207175325"
Sep 14 13:00:16.731927 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:16.731882    2557 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms" podUID="2a94c6ba-2e5a-4dd5-9530-f146816e158c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 14 13:00:20.761907 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:20.761869    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8565875799-kvq4d" podUID="f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e" containerName="registry" containerID="cri-o://b5fc440de29b1ca4801675fdf182946af310d90e3ef478b2770308bbec07e942" gracePeriod=30
Sep 14 13:00:20.994006 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:20.993983    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 13:00:21.105325 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.105258    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-certificates\") pod \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") "
Sep 14 13:00:21.105325 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.105321    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-ca-trust-extracted\") pod \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") "
Sep 14 13:00:21.105490 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.105356    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls\") pod \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") "
Sep 14 13:00:21.106339 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.105762    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-installation-pull-secrets\") pod \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") "
Sep 14 13:00:21.106339 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.105839    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-trusted-ca\") pod \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") "
Sep 14 13:00:21.106339 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.105897    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmx2q\" (UniqueName: \"kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-kube-api-access-hmx2q\") pod \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") "
Sep 14 13:00:21.106339 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.105937    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-bound-sa-token\") pod \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") "
Sep 14 13:00:21.106339 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.106100    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e" (UID: "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:00:21.106656 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.106413    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e" (UID: "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:00:21.107016 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.106993    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-image-registry-private-configuration\") pod \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\" (UID: \"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e\") "
Sep 14 13:00:21.107705 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.107677    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-trusted-ca\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:00:21.107819 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.107718    2557 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-certificates\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:00:21.109422 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.109263    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e" (UID: "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:00:21.110718 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.110410    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e" (UID: "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:00:21.112012 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.111985    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-kube-api-access-hmx2q" (OuterVolumeSpecName: "kube-api-access-hmx2q") pod "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e" (UID: "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e"). InnerVolumeSpecName "kube-api-access-hmx2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:00:21.112372 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.112339    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e" (UID: "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:00:21.112710 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.112682    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e" (UID: "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:00:21.116229 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.116207    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e" (UID: "f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:00:21.208503 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.208459    2557 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-ca-trust-extracted\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:00:21.208503 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.208498    2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-registry-tls\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:00:21.208503 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.208508    2557 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-installation-pull-secrets\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:00:21.208503 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.208516    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmx2q\" (UniqueName: \"kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-kube-api-access-hmx2q\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:00:21.208731 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.208525    2557 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-bound-sa-token\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:00:21.208731 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.208534    2557 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e-image-registry-private-configuration\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:00:21.783317 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.783284    2557 generic.go:356] "Generic (PLEG): container finished" podID="f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e" containerID="b5fc440de29b1ca4801675fdf182946af310d90e3ef478b2770308bbec07e942" exitCode=0
Sep 14 13:00:21.783706 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.783332    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8565875799-kvq4d"
Sep 14 13:00:21.783706 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.783335    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8565875799-kvq4d" event={"ID":"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e","Type":"ContainerDied","Data":"b5fc440de29b1ca4801675fdf182946af310d90e3ef478b2770308bbec07e942"}
Sep 14 13:00:21.783706 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.783468    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8565875799-kvq4d" event={"ID":"f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e","Type":"ContainerDied","Data":"7ab1ad50599bd53213f0a091864fc36a3fcbe524f586b7eccee86ba3c293d8d7"}
Sep 14 13:00:21.783706 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.783500    2557 scope.go:117] "RemoveContainer" containerID="b5fc440de29b1ca4801675fdf182946af310d90e3ef478b2770308bbec07e942"
Sep 14 13:00:21.792661 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.792642    2557 scope.go:117] "RemoveContainer" containerID="b5fc440de29b1ca4801675fdf182946af310d90e3ef478b2770308bbec07e942"
Sep 14 13:00:21.792956 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:00:21.792936    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5fc440de29b1ca4801675fdf182946af310d90e3ef478b2770308bbec07e942\": container with ID starting with b5fc440de29b1ca4801675fdf182946af310d90e3ef478b2770308bbec07e942 not found: ID does not exist" containerID="b5fc440de29b1ca4801675fdf182946af310d90e3ef478b2770308bbec07e942"
Sep 14 13:00:21.793047 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.792963    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5fc440de29b1ca4801675fdf182946af310d90e3ef478b2770308bbec07e942"} err="failed to get container status \"b5fc440de29b1ca4801675fdf182946af310d90e3ef478b2770308bbec07e942\": rpc error: code = NotFound desc = could not find container \"b5fc440de29b1ca4801675fdf182946af310d90e3ef478b2770308bbec07e942\": container with ID starting with b5fc440de29b1ca4801675fdf182946af310d90e3ef478b2770308bbec07e942 not found: ID does not exist"
Sep 14 13:00:21.806910 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.806892    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8565875799-kvq4d"]
Sep 14 13:00:21.810561 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:21.810540    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8565875799-kvq4d"]
Sep 14 13:00:22.047268 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:22.047196    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e" path="/var/lib/kubelet/pods/f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e/volumes"
Sep 14 13:00:23.257365 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:23.257338    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jmcw8_d44e13a7-7e76-4eee-873f-68bf4066ac29/serve-healthcheck-canary/0.log"
Sep 14 13:00:26.731969 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:26.731924    2557 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms" podUID="2a94c6ba-2e5a-4dd5-9530-f146816e158c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 14 13:00:36.731182 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:36.731139    2557 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms" podUID="2a94c6ba-2e5a-4dd5-9530-f146816e158c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 14 13:00:36.731661 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:36.731227    2557 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms"
Sep 14 13:00:36.731949 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:36.731841    2557 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"c11ffbf297c8321eb8f403b7e72211c6c3e40ad0e902e144fa2af71a0ca3891a"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 14 13:00:36.731949 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:36.731897    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms" podUID="2a94c6ba-2e5a-4dd5-9530-f146816e158c" containerName="service-proxy" containerID="cri-o://c11ffbf297c8321eb8f403b7e72211c6c3e40ad0e902e144fa2af71a0ca3891a" gracePeriod=30
Sep 14 13:00:37.838776 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:37.838731    2557 generic.go:356] "Generic (PLEG): container finished" podID="2a94c6ba-2e5a-4dd5-9530-f146816e158c" containerID="c11ffbf297c8321eb8f403b7e72211c6c3e40ad0e902e144fa2af71a0ca3891a" exitCode=2
Sep 14 13:00:37.838776 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:37.838771    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms" event={"ID":"2a94c6ba-2e5a-4dd5-9530-f146816e158c","Type":"ContainerDied","Data":"c11ffbf297c8321eb8f403b7e72211c6c3e40ad0e902e144fa2af71a0ca3891a"}
Sep 14 13:00:37.839190 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:00:37.838821    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df7df9c64-kzgms" event={"ID":"2a94c6ba-2e5a-4dd5-9530-f146816e158c","Type":"ContainerStarted","Data":"2d0204513028e3113d7fc9b1441a0b6cf610b0e0270d71f28d4274dd3a93bb35"}
Sep 14 13:03:01.989134 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:03:01.989108    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwqnf_62d953d6-3b86-4c95-9a47-9579d88993a7/ovn-acl-logging/0.log"
Sep 14 13:03:01.991664 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:03:01.991646    2557 kubelet.go:1648] "Image garbage collection succeeded"
Sep 14 13:03:01.999807 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:03:01.999780    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwqnf_62d953d6-3b86-4c95-9a47-9579d88993a7/ovn-acl-logging/0.log"
Sep 14 13:04:33.222867 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:33.222837    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jbkrl"]
Sep 14 13:04:33.223317 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:33.223108    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e" containerName="registry"
Sep 14 13:04:33.223317 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:33.223119    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e" containerName="registry"
Sep 14 13:04:33.223317 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:33.223157    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5bf9a24-6ac3-402d-b91a-f0fed1f13a3e" containerName="registry"
Sep 14 13:04:33.225783 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:33.225767    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jbkrl"
Sep 14 13:04:33.230557 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:33.230533    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 14 13:04:33.231935 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:33.231913    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-zfzgj\""
Sep 14 13:04:33.232038 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:33.231948    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 14 13:04:33.242133 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:33.242108    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jbkrl"]
Sep 14 13:04:33.348030 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:33.347993    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh8jx\" (UniqueName: \"kubernetes.io/projected/dc1fed35-c3ce-4962-aadc-1529061210d6-kube-api-access-lh8jx\") pod \"openshift-lws-operator-fd8ccff4c-jbkrl\" (UID: \"dc1fed35-c3ce-4962-aadc-1529061210d6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jbkrl"
Sep 14 13:04:33.348209 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:33.348043    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc1fed35-c3ce-4962-aadc-1529061210d6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jbkrl\" (UID: \"dc1fed35-c3ce-4962-aadc-1529061210d6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jbkrl"
Sep 14 13:04:33.448359 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:33.448305    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc1fed35-c3ce-4962-aadc-1529061210d6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jbkrl\" (UID: \"dc1fed35-c3ce-4962-aadc-1529061210d6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jbkrl"
Sep 14 13:04:33.448500 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:33.448391    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8jx\" (UniqueName: \"kubernetes.io/projected/dc1fed35-c3ce-4962-aadc-1529061210d6-kube-api-access-lh8jx\") pod \"openshift-lws-operator-fd8ccff4c-jbkrl\" (UID: \"dc1fed35-c3ce-4962-aadc-1529061210d6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jbkrl"
Sep 14 13:04:33.448777 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:33.448730    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc1fed35-c3ce-4962-aadc-1529061210d6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jbkrl\" (UID: \"dc1fed35-c3ce-4962-aadc-1529061210d6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jbkrl"
Sep 14 13:04:33.457938 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:33.457912    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh8jx\" (UniqueName: \"kubernetes.io/projected/dc1fed35-c3ce-4962-aadc-1529061210d6-kube-api-access-lh8jx\") pod \"openshift-lws-operator-fd8ccff4c-jbkrl\" (UID: \"dc1fed35-c3ce-4962-aadc-1529061210d6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jbkrl"
Sep 14 13:04:33.533968 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:33.533874    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jbkrl"
Sep 14 13:04:33.803685 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:33.803662    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jbkrl"]
Sep 14 13:04:33.805927 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:04:33.805900    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc1fed35_c3ce_4962_aadc_1529061210d6.slice/crio-38155168c81f508f61c3957a4912bf76f5a8bd54df61b9655f80a116b3be5752 WatchSource:0}: Error finding container 38155168c81f508f61c3957a4912bf76f5a8bd54df61b9655f80a116b3be5752: Status 404 returned error can't find the container with id 38155168c81f508f61c3957a4912bf76f5a8bd54df61b9655f80a116b3be5752
Sep 14 13:04:33.807579 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:33.807561    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 13:04:34.598631 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:34.598600    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jbkrl" event={"ID":"dc1fed35-c3ce-4962-aadc-1529061210d6","Type":"ContainerStarted","Data":"38155168c81f508f61c3957a4912bf76f5a8bd54df61b9655f80a116b3be5752"}
Sep 14 13:04:37.609916 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:37.609872    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jbkrl" event={"ID":"dc1fed35-c3ce-4962-aadc-1529061210d6","Type":"ContainerStarted","Data":"8cff6c076358f36ffb542a6aa2299940a40c8dcb56e3ff4352a950289387f9aa"}
Sep 14 13:04:37.625576 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:37.625529    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jbkrl" podStartSLOduration=1.762672518 podStartE2EDuration="4.625512616s" totalImagesPullingTime="2.862840098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:04:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:04:37.624513885 +0000 UTC m=+396.059189405" watchObservedRunningTime="2026-09-14 13:04:37.625512616 +0000 UTC m=+396.060188134"
Sep 14 13:04:53.066728 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.066693    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8"]
Sep 14 13:04:53.073173 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.073151    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8"
Sep 14 13:04:53.076270 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.076250    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 14 13:04:53.076816 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.076791    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 14 13:04:53.076930 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.076815    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 14 13:04:53.077454 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.077434    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-bdfph\""
Sep 14 13:04:53.081307 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.081283    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 14 13:04:53.087045 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.087024    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8"]
Sep 14 13:04:53.182649 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.182619    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca27e799-89fb-4e04-9882-68c2fba75176-tmp\") pod \"opendatahub-operator-controller-manager-697cf97c89-j6tj8\" (UID: \"ca27e799-89fb-4e04-9882-68c2fba75176\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8"
Sep 14 13:04:53.182649 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.182652    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtkcs\" (UniqueName: \"kubernetes.io/projected/ca27e799-89fb-4e04-9882-68c2fba75176-kube-api-access-vtkcs\") pod \"opendatahub-operator-controller-manager-697cf97c89-j6tj8\" (UID: \"ca27e799-89fb-4e04-9882-68c2fba75176\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8"
Sep 14 13:04:53.182887 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.182681    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ca27e799-89fb-4e04-9882-68c2fba75176-webhook-cert\") pod \"opendatahub-operator-controller-manager-697cf97c89-j6tj8\" (UID: \"ca27e799-89fb-4e04-9882-68c2fba75176\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8"
Sep 14 13:04:53.182887 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.182784    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ca27e799-89fb-4e04-9882-68c2fba75176-manifests\") pod \"opendatahub-operator-controller-manager-697cf97c89-j6tj8\" (UID: \"ca27e799-89fb-4e04-9882-68c2fba75176\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8"
Sep 14 13:04:53.182887 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.182826    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ca27e799-89fb-4e04-9882-68c2fba75176-apiservice-cert\") pod \"opendatahub-operator-controller-manager-697cf97c89-j6tj8\" (UID: \"ca27e799-89fb-4e04-9882-68c2fba75176\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8"
Sep 14 13:04:53.284097 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.284063    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca27e799-89fb-4e04-9882-68c2fba75176-tmp\") pod \"opendatahub-operator-controller-manager-697cf97c89-j6tj8\" (UID: \"ca27e799-89fb-4e04-9882-68c2fba75176\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8"
Sep 14 13:04:53.284097 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.284102    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtkcs\" (UniqueName: \"kubernetes.io/projected/ca27e799-89fb-4e04-9882-68c2fba75176-kube-api-access-vtkcs\") pod \"opendatahub-operator-controller-manager-697cf97c89-j6tj8\" (UID: \"ca27e799-89fb-4e04-9882-68c2fba75176\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8"
Sep 14 13:04:53.284342 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.284137    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ca27e799-89fb-4e04-9882-68c2fba75176-webhook-cert\") pod \"opendatahub-operator-controller-manager-697cf97c89-j6tj8\" (UID: \"ca27e799-89fb-4e04-9882-68c2fba75176\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8"
Sep 14 13:04:53.284342 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.284179    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ca27e799-89fb-4e04-9882-68c2fba75176-manifests\") pod \"opendatahub-operator-controller-manager-697cf97c89-j6tj8\" (UID: \"ca27e799-89fb-4e04-9882-68c2fba75176\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8"
Sep 14 13:04:53.284342 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.284212    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ca27e799-89fb-4e04-9882-68c2fba75176-apiservice-cert\") pod \"opendatahub-operator-controller-manager-697cf97c89-j6tj8\" (UID: \"ca27e799-89fb-4e04-9882-68c2fba75176\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8"
Sep 14 13:04:53.284561 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.284540    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ca27e799-89fb-4e04-9882-68c2fba75176-manifests\") pod \"opendatahub-operator-controller-manager-697cf97c89-j6tj8\" (UID: \"ca27e799-89fb-4e04-9882-68c2fba75176\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8"
Sep 14 13:04:53.286404 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.286384    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca27e799-89fb-4e04-9882-68c2fba75176-tmp\") pod \"opendatahub-operator-controller-manager-697cf97c89-j6tj8\" (UID: \"ca27e799-89fb-4e04-9882-68c2fba75176\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8"
Sep 14 13:04:53.286701 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.286681    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ca27e799-89fb-4e04-9882-68c2fba75176-apiservice-cert\") pod \"opendatahub-operator-controller-manager-697cf97c89-j6tj8\" (UID: \"ca27e799-89fb-4e04-9882-68c2fba75176\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8"
Sep 14 13:04:53.286798 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.286775    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ca27e799-89fb-4e04-9882-68c2fba75176-webhook-cert\") pod \"opendatahub-operator-controller-manager-697cf97c89-j6tj8\" (UID: \"ca27e799-89fb-4e04-9882-68c2fba75176\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8"
Sep 14 13:04:53.293435 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.293408    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtkcs\" (UniqueName: \"kubernetes.io/projected/ca27e799-89fb-4e04-9882-68c2fba75176-kube-api-access-vtkcs\") pod \"opendatahub-operator-controller-manager-697cf97c89-j6tj8\" (UID: \"ca27e799-89fb-4e04-9882-68c2fba75176\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8"
Sep 14 13:04:53.388421 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.388334    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8"
Sep 14 13:04:53.668783 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:53.668691    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8"]
Sep 14 13:04:53.671802 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:04:53.671773    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca27e799_89fb_4e04_9882_68c2fba75176.slice/crio-ab73899c9e3178850305b4ed8b70c7c505e73b0f6cfffada15a300b0cc5a6298 WatchSource:0}: Error finding container ab73899c9e3178850305b4ed8b70c7c505e73b0f6cfffada15a300b0cc5a6298: Status 404 returned error can't find the container with id ab73899c9e3178850305b4ed8b70c7c505e73b0f6cfffada15a300b0cc5a6298
Sep 14 13:04:54.668713 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:54.668635    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8" event={"ID":"ca27e799-89fb-4e04-9882-68c2fba75176","Type":"ContainerStarted","Data":"ab73899c9e3178850305b4ed8b70c7c505e73b0f6cfffada15a300b0cc5a6298"}
Sep 14 13:04:56.677146 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:56.677116    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8" event={"ID":"ca27e799-89fb-4e04-9882-68c2fba75176","Type":"ContainerStarted","Data":"a2d6fe0efa793fd9bb89e8bee0c2484810c83d9633b933ac45cb066fde42a369"}
Sep 14 13:04:57.681594 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:57.681564    2557 generic.go:356] "Generic (PLEG): container finished" podID="ca27e799-89fb-4e04-9882-68c2fba75176" containerID="a2d6fe0efa793fd9bb89e8bee0c2484810c83d9633b933ac45cb066fde42a369" exitCode=0
Sep 14 13:04:57.682108 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:57.681650    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8" event={"ID":"ca27e799-89fb-4e04-9882-68c2fba75176","Type":"ContainerDied","Data":"a2d6fe0efa793fd9bb89e8bee0c2484810c83d9633b933ac45cb066fde42a369"}
Sep 14 13:04:58.688060 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:58.688018    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8" event={"ID":"ca27e799-89fb-4e04-9882-68c2fba75176","Type":"ContainerStarted","Data":"0639fdb881559c2c939ac9dd1c5f6dc02cda0b10658f56b1f5e8ad65a2d010e8"}
Sep 14 13:04:58.688460 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:58.688111    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8"
Sep 14 13:04:58.710935 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:04:58.710879    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8" podStartSLOduration=1.39161692 podStartE2EDuration="5.710861628s" totalImagesPullingTime="3.291687865s" totalInitContainerRuntime="1.027556843s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:04:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:04:58.709498103 +0000 UTC m=+417.144173624" watchObservedRunningTime="2026-09-14 13:04:58.710861628 +0000 UTC m=+417.145537147"
Sep 14 13:05:09.694260 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:09.694231    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-j6tj8"
Sep 14 13:05:23.416162 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:23.416132    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7f6f8d7966-fnc7x"]
Sep 14 13:05:23.424522 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:23.424491    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f6f8d7966-fnc7x"
Sep 14 13:05:23.428370 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:23.428345    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 14 13:05:23.428488 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:23.428392    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 14 13:05:23.429254 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:23.429226    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 14 13:05:23.429254 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:23.429226    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 14 13:05:23.429424 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:23.429298    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7p42f\""
Sep 14 13:05:23.437960 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:23.437938    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f6f8d7966-fnc7x"]
Sep 14 13:05:23.492307 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:23.492287    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1729b2a5-0803-4319-97ae-00eb46f0016c-tls-certs\") pod \"kube-auth-proxy-7f6f8d7966-fnc7x\" (UID: \"1729b2a5-0803-4319-97ae-00eb46f0016c\") " pod="openshift-ingress/kube-auth-proxy-7f6f8d7966-fnc7x"
Sep 14 13:05:23.492421 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:23.492325    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnxbc\" (UniqueName: \"kubernetes.io/projected/1729b2a5-0803-4319-97ae-00eb46f0016c-kube-api-access-qnxbc\") pod \"kube-auth-proxy-7f6f8d7966-fnc7x\" (UID: \"1729b2a5-0803-4319-97ae-00eb46f0016c\") " pod="openshift-ingress/kube-auth-proxy-7f6f8d7966-fnc7x"
Sep 14 13:05:23.492421 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:23.492359    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1729b2a5-0803-4319-97ae-00eb46f0016c-tmp\") pod \"kube-auth-proxy-7f6f8d7966-fnc7x\" (UID: \"1729b2a5-0803-4319-97ae-00eb46f0016c\") " pod="openshift-ingress/kube-auth-proxy-7f6f8d7966-fnc7x"
Sep 14 13:05:23.593459 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:23.593434    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1729b2a5-0803-4319-97ae-00eb46f0016c-tls-certs\") pod \"kube-auth-proxy-7f6f8d7966-fnc7x\" (UID: \"1729b2a5-0803-4319-97ae-00eb46f0016c\") " pod="openshift-ingress/kube-auth-proxy-7f6f8d7966-fnc7x"
Sep 14 13:05:23.593583 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:23.593476    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnxbc\" (UniqueName: \"kubernetes.io/projected/1729b2a5-0803-4319-97ae-00eb46f0016c-kube-api-access-qnxbc\") pod \"kube-auth-proxy-7f6f8d7966-fnc7x\" (UID: \"1729b2a5-0803-4319-97ae-00eb46f0016c\") " pod="openshift-ingress/kube-auth-proxy-7f6f8d7966-fnc7x"
Sep 14 13:05:23.593583 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:23.593503    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1729b2a5-0803-4319-97ae-00eb46f0016c-tmp\") pod \"kube-auth-proxy-7f6f8d7966-fnc7x\" (UID: \"1729b2a5-0803-4319-97ae-00eb46f0016c\") " pod="openshift-ingress/kube-auth-proxy-7f6f8d7966-fnc7x"
Sep 14 13:05:23.595726 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:23.595697    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1729b2a5-0803-4319-97ae-00eb46f0016c-tmp\") pod \"kube-auth-proxy-7f6f8d7966-fnc7x\" (UID: \"1729b2a5-0803-4319-97ae-00eb46f0016c\") " pod="openshift-ingress/kube-auth-proxy-7f6f8d7966-fnc7x"
Sep 14 13:05:23.595915 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:23.595896    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1729b2a5-0803-4319-97ae-00eb46f0016c-tls-certs\") pod \"kube-auth-proxy-7f6f8d7966-fnc7x\" (UID: \"1729b2a5-0803-4319-97ae-00eb46f0016c\") " pod="openshift-ingress/kube-auth-proxy-7f6f8d7966-fnc7x"
Sep 14 13:05:23.604666 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:23.604649    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnxbc\" (UniqueName: \"kubernetes.io/projected/1729b2a5-0803-4319-97ae-00eb46f0016c-kube-api-access-qnxbc\") pod \"kube-auth-proxy-7f6f8d7966-fnc7x\" (UID: \"1729b2a5-0803-4319-97ae-00eb46f0016c\") " pod="openshift-ingress/kube-auth-proxy-7f6f8d7966-fnc7x"
Sep 14 13:05:23.733992 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:23.733971    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f6f8d7966-fnc7x"
Sep 14 13:05:24.006528 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.006452    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f6f8d7966-fnc7x"]
Sep 14 13:05:24.010617 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:05:24.010590    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1729b2a5_0803_4319_97ae_00eb46f0016c.slice/crio-40643c2035c9fd8436b465e8e679d5074c83cfc073740cbed0032a1f630e1db9 WatchSource:0}: Error finding container 40643c2035c9fd8436b465e8e679d5074c83cfc073740cbed0032a1f630e1db9: Status 404 returned error can't find the container with id 40643c2035c9fd8436b465e8e679d5074c83cfc073740cbed0032a1f630e1db9
Sep 14 13:05:24.611805 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.611633    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"]
Sep 14 13:05:24.615110 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.615087    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:24.618035 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.617954    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 14 13:05:24.618035 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.617954    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 14 13:05:24.618035 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.618016    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 14 13:05:24.618232 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.617963    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 14 13:05:24.618232 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.618157    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-27rzz\""
Sep 14 13:05:24.628544 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.628520    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"]
Sep 14 13:05:24.701452 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.701422    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9tnz\" (UniqueName: \"kubernetes.io/projected/864604c0-07c8-4a82-8b5f-092e5d70457c-kube-api-access-n9tnz\") pod \"istiod-openshift-gateway-7b7995d5cd-6f28p\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:24.701712 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.701476    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/864604c0-07c8-4a82-8b5f-092e5d70457c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6f28p\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:24.701712 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.701529    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/864604c0-07c8-4a82-8b5f-092e5d70457c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6f28p\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:24.701712 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.701553    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6f28p\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:24.701712 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.701605    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6f28p\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:24.701712 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.701638    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6f28p\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:24.701712 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.701686    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6f28p\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:24.783431 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.783364    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f6f8d7966-fnc7x" event={"ID":"1729b2a5-0803-4319-97ae-00eb46f0016c","Type":"ContainerStarted","Data":"40643c2035c9fd8436b465e8e679d5074c83cfc073740cbed0032a1f630e1db9"}
Sep 14 13:05:24.802033 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.802003    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/864604c0-07c8-4a82-8b5f-092e5d70457c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6f28p\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:24.802149 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.802047    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6f28p\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:24.802149 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.802086    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6f28p\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:24.802149 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.802115    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6f28p\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:24.802332 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.802312    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6f28p\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:24.802396 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.802380    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9tnz\" (UniqueName: \"kubernetes.io/projected/864604c0-07c8-4a82-8b5f-092e5d70457c-kube-api-access-n9tnz\") pod \"istiod-openshift-gateway-7b7995d5cd-6f28p\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:24.802454 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.802424    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/864604c0-07c8-4a82-8b5f-092e5d70457c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6f28p\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:24.803055 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.802981    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6f28p\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:24.805076 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.805052    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/864604c0-07c8-4a82-8b5f-092e5d70457c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6f28p\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:24.805461 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.805437    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/864604c0-07c8-4a82-8b5f-092e5d70457c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6f28p\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:24.805538 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.805495    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6f28p\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:24.806218 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.806195    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6f28p\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:24.810713 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.810672    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9tnz\" (UniqueName: \"kubernetes.io/projected/864604c0-07c8-4a82-8b5f-092e5d70457c-kube-api-access-n9tnz\") pod \"istiod-openshift-gateway-7b7995d5cd-6f28p\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:24.811049 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.811026    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6f28p\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:24.927438 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:24.927383    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:25.216322 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:25.216251    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"]
Sep 14 13:05:25.887353 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:05:25.887320    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864604c0_07c8_4a82_8b5f_092e5d70457c.slice/crio-75c10f3b0b475ab1b8ee68f405be4efd62b293181e20214a55aa67dcc004ddf9 WatchSource:0}: Error finding container 75c10f3b0b475ab1b8ee68f405be4efd62b293181e20214a55aa67dcc004ddf9: Status 404 returned error can't find the container with id 75c10f3b0b475ab1b8ee68f405be4efd62b293181e20214a55aa67dcc004ddf9
Sep 14 13:05:26.795877 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:26.795833    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p" event={"ID":"864604c0-07c8-4a82-8b5f-092e5d70457c","Type":"ContainerStarted","Data":"75c10f3b0b475ab1b8ee68f405be4efd62b293181e20214a55aa67dcc004ddf9"}
Sep 14 13:05:26.797923 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:26.797892    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f6f8d7966-fnc7x" event={"ID":"1729b2a5-0803-4319-97ae-00eb46f0016c","Type":"ContainerStarted","Data":"3acff8145e178cd9cbf5fcff16a93c872b3173b6b59a44292426eb9bd53ed2f3"}
Sep 14 13:05:26.815217 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:26.815126    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7f6f8d7966-fnc7x" podStartSLOduration=1.265172896 podStartE2EDuration="3.815109532s" totalImagesPullingTime="2.549936636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:05:26.814351052 +0000 UTC m=+445.249026584" watchObservedRunningTime="2026-09-14 13:05:26.815109532 +0000 UTC m=+445.249785051"
Sep 14 13:05:28.252652 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:28.252615    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:05:28.253045 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:28.252699    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:05:28.806699 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:28.806668    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p" event={"ID":"864604c0-07c8-4a82-8b5f-092e5d70457c","Type":"ContainerStarted","Data":"f134ba2cfee025dcec52c47f8d85b4fa2219d1d1b6d340d670ac842db113fb1b"}
Sep 14 13:05:28.806869 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:28.806791    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:28.830531 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:28.830476    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p" podStartSLOduration=2.467300671 podStartE2EDuration="4.830456685s" totalImagesPullingTime="2.363156014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:05:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:05:28.827844499 +0000 UTC m=+447.262520021" watchObservedRunningTime="2026-09-14 13:05:28.830456685 +0000 UTC m=+447.265132205"
Sep 14 13:05:29.813247 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:29.813207    2557 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-6f28p container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 13:05:29.813780 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:29.813274    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p" podUID="864604c0-07c8-4a82-8b5f-092e5d70457c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 13:05:32.812552 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:32.812518    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:34.535579 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:34.535543    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xfqw6"]
Sep 14 13:05:34.542959 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:34.542937    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-xfqw6"
Sep 14 13:05:34.545688 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:34.545672    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 14 13:05:34.551024 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:34.550999    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xfqw6"]
Sep 14 13:05:34.579038 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:34.579016    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kksl\" (UniqueName: \"kubernetes.io/projected/a82fb799-541d-4bbc-9650-c22301c0bb25-kube-api-access-4kksl\") pod \"model-serving-api-54d89bd5fc-xfqw6\" (UID: \"a82fb799-541d-4bbc-9650-c22301c0bb25\") " pod="opendatahub/model-serving-api-54d89bd5fc-xfqw6"
Sep 14 13:05:34.579162 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:34.579047    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a82fb799-541d-4bbc-9650-c22301c0bb25-tls-certs\") pod \"model-serving-api-54d89bd5fc-xfqw6\" (UID: \"a82fb799-541d-4bbc-9650-c22301c0bb25\") " pod="opendatahub/model-serving-api-54d89bd5fc-xfqw6"
Sep 14 13:05:34.680063 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:34.680027    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kksl\" (UniqueName: \"kubernetes.io/projected/a82fb799-541d-4bbc-9650-c22301c0bb25-kube-api-access-4kksl\") pod \"model-serving-api-54d89bd5fc-xfqw6\" (UID: \"a82fb799-541d-4bbc-9650-c22301c0bb25\") " pod="opendatahub/model-serving-api-54d89bd5fc-xfqw6"
Sep 14 13:05:34.680063 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:34.680065    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a82fb799-541d-4bbc-9650-c22301c0bb25-tls-certs\") pod \"model-serving-api-54d89bd5fc-xfqw6\" (UID: \"a82fb799-541d-4bbc-9650-c22301c0bb25\") " pod="opendatahub/model-serving-api-54d89bd5fc-xfqw6"
Sep 14 13:05:34.680280 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:05:34.680174    2557 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 14 13:05:34.680280 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:05:34.680237    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a82fb799-541d-4bbc-9650-c22301c0bb25-tls-certs podName:a82fb799-541d-4bbc-9650-c22301c0bb25 nodeName:}" failed. No retries permitted until 2026-09-14 13:05:35.180220103 +0000 UTC m=+453.614895599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a82fb799-541d-4bbc-9650-c22301c0bb25-tls-certs") pod "model-serving-api-54d89bd5fc-xfqw6" (UID: "a82fb799-541d-4bbc-9650-c22301c0bb25") : secret "model-serving-api-tls" not found
Sep 14 13:05:34.691631 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:34.691598    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kksl\" (UniqueName: \"kubernetes.io/projected/a82fb799-541d-4bbc-9650-c22301c0bb25-kube-api-access-4kksl\") pod \"model-serving-api-54d89bd5fc-xfqw6\" (UID: \"a82fb799-541d-4bbc-9650-c22301c0bb25\") " pod="opendatahub/model-serving-api-54d89bd5fc-xfqw6"
Sep 14 13:05:35.183686 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:35.183653    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a82fb799-541d-4bbc-9650-c22301c0bb25-tls-certs\") pod \"model-serving-api-54d89bd5fc-xfqw6\" (UID: \"a82fb799-541d-4bbc-9650-c22301c0bb25\") " pod="opendatahub/model-serving-api-54d89bd5fc-xfqw6"
Sep 14 13:05:35.186186 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:35.186161    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a82fb799-541d-4bbc-9650-c22301c0bb25-tls-certs\") pod \"model-serving-api-54d89bd5fc-xfqw6\" (UID: \"a82fb799-541d-4bbc-9650-c22301c0bb25\") " pod="opendatahub/model-serving-api-54d89bd5fc-xfqw6"
Sep 14 13:05:35.454165 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:35.454133    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-xfqw6"
Sep 14 13:05:35.723103 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:35.722907    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xfqw6"]
Sep 14 13:05:35.725488 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:05:35.725460    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82fb799_541d_4bbc_9650_c22301c0bb25.slice/crio-2459024b70bcbbabd37d89788ebf1fbf5ab3e2008c22512cca8072af8a204cba WatchSource:0}: Error finding container 2459024b70bcbbabd37d89788ebf1fbf5ab3e2008c22512cca8072af8a204cba: Status 404 returned error can't find the container with id 2459024b70bcbbabd37d89788ebf1fbf5ab3e2008c22512cca8072af8a204cba
Sep 14 13:05:35.834540 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:35.834508    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-xfqw6" event={"ID":"a82fb799-541d-4bbc-9650-c22301c0bb25","Type":"ContainerStarted","Data":"2459024b70bcbbabd37d89788ebf1fbf5ab3e2008c22512cca8072af8a204cba"}
Sep 14 13:05:37.843986 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:37.843949    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-xfqw6" event={"ID":"a82fb799-541d-4bbc-9650-c22301c0bb25","Type":"ContainerStarted","Data":"26033fb461623669e100c4b7a07bd60f8b6ab7042c99dc8665e4c12153e8342c"}
Sep 14 13:05:37.844344 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:37.844089    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-xfqw6"
Sep 14 13:05:37.862281 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:37.862223    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-xfqw6" podStartSLOduration=2.211487191 podStartE2EDuration="3.862206012s" totalImagesPullingTime="1.650718821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:05:37.861959927 +0000 UTC m=+456.296635446" watchObservedRunningTime="2026-09-14 13:05:37.862206012 +0000 UTC m=+456.296881532"
Sep 14 13:05:43.166439 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.166408    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gmncm"]
Sep 14 13:05:43.169797 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.169774    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gmncm"
Sep 14 13:05:43.173038 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.173015    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-6f9h5\""
Sep 14 13:05:43.173137 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.173090    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 14 13:05:43.179044 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.179019    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-fpq4w"]
Sep 14 13:05:43.182043 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.182023    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpq4w"
Sep 14 13:05:43.183149 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.183130    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gmncm"]
Sep 14 13:05:43.184719 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.184699    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 14 13:05:43.185264 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.185246    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-dlqzq\""
Sep 14 13:05:43.202465 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.202441    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-fpq4w"]
Sep 14 13:05:43.243693 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.243668    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fd5deac-72f2-4b25-9db8-82735b3ad959-cert\") pod \"kserve-controller-manager-d6f4b65b8-gmncm\" (UID: \"6fd5deac-72f2-4b25-9db8-82735b3ad959\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gmncm"
Sep 14 13:05:43.243809 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.243702    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcszp\" (UniqueName: \"kubernetes.io/projected/6fd5deac-72f2-4b25-9db8-82735b3ad959-kube-api-access-qcszp\") pod \"kserve-controller-manager-d6f4b65b8-gmncm\" (UID: \"6fd5deac-72f2-4b25-9db8-82735b3ad959\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gmncm"
Sep 14 13:05:43.243860 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.243810    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/120fe86c-4cf8-488d-bdb3-0c5a120d35dd-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fpq4w\" (UID: \"120fe86c-4cf8-488d-bdb3-0c5a120d35dd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpq4w"
Sep 14 13:05:43.243860 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.243842    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4bht\" (UniqueName: \"kubernetes.io/projected/120fe86c-4cf8-488d-bdb3-0c5a120d35dd-kube-api-access-n4bht\") pod \"llmisvc-controller-manager-777c6bd7d5-fpq4w\" (UID: \"120fe86c-4cf8-488d-bdb3-0c5a120d35dd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpq4w"
Sep 14 13:05:43.344660 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.344635    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/120fe86c-4cf8-488d-bdb3-0c5a120d35dd-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fpq4w\" (UID: \"120fe86c-4cf8-488d-bdb3-0c5a120d35dd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpq4w"
Sep 14 13:05:43.344778 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.344668    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4bht\" (UniqueName: \"kubernetes.io/projected/120fe86c-4cf8-488d-bdb3-0c5a120d35dd-kube-api-access-n4bht\") pod \"llmisvc-controller-manager-777c6bd7d5-fpq4w\" (UID: \"120fe86c-4cf8-488d-bdb3-0c5a120d35dd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpq4w"
Sep 14 13:05:43.344778 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.344690    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fd5deac-72f2-4b25-9db8-82735b3ad959-cert\") pod \"kserve-controller-manager-d6f4b65b8-gmncm\" (UID: \"6fd5deac-72f2-4b25-9db8-82735b3ad959\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gmncm"
Sep 14 13:05:43.344866 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:05:43.344795    2557 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 14 13:05:43.344866 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:05:43.344853    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fd5deac-72f2-4b25-9db8-82735b3ad959-cert podName:6fd5deac-72f2-4b25-9db8-82735b3ad959 nodeName:}" failed. No retries permitted until 2026-09-14 13:05:43.844832918 +0000 UTC m=+462.279508432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6fd5deac-72f2-4b25-9db8-82735b3ad959-cert") pod "kserve-controller-manager-d6f4b65b8-gmncm" (UID: "6fd5deac-72f2-4b25-9db8-82735b3ad959") : secret "kserve-webhook-server-cert" not found
Sep 14 13:05:43.344952 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.344877    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcszp\" (UniqueName: \"kubernetes.io/projected/6fd5deac-72f2-4b25-9db8-82735b3ad959-kube-api-access-qcszp\") pod \"kserve-controller-manager-d6f4b65b8-gmncm\" (UID: \"6fd5deac-72f2-4b25-9db8-82735b3ad959\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gmncm"
Sep 14 13:05:43.347071 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.347053    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/120fe86c-4cf8-488d-bdb3-0c5a120d35dd-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fpq4w\" (UID: \"120fe86c-4cf8-488d-bdb3-0c5a120d35dd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpq4w"
Sep 14 13:05:43.358460 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.358428    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4bht\" (UniqueName: \"kubernetes.io/projected/120fe86c-4cf8-488d-bdb3-0c5a120d35dd-kube-api-access-n4bht\") pod \"llmisvc-controller-manager-777c6bd7d5-fpq4w\" (UID: \"120fe86c-4cf8-488d-bdb3-0c5a120d35dd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpq4w"
Sep 14 13:05:43.359222 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.359205    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcszp\" (UniqueName: \"kubernetes.io/projected/6fd5deac-72f2-4b25-9db8-82735b3ad959-kube-api-access-qcszp\") pod \"kserve-controller-manager-d6f4b65b8-gmncm\" (UID: \"6fd5deac-72f2-4b25-9db8-82735b3ad959\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gmncm"
Sep 14 13:05:43.491710 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.491685    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpq4w"
Sep 14 13:05:43.759521 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.759363    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-fpq4w"]
Sep 14 13:05:43.761859 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:05:43.761836    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod120fe86c_4cf8_488d_bdb3_0c5a120d35dd.slice/crio-e5e49aeb458388f120ac3d33a837111073e307d79bb782ef9bbf3e7288a70c7c WatchSource:0}: Error finding container e5e49aeb458388f120ac3d33a837111073e307d79bb782ef9bbf3e7288a70c7c: Status 404 returned error can't find the container with id e5e49aeb458388f120ac3d33a837111073e307d79bb782ef9bbf3e7288a70c7c
Sep 14 13:05:43.848624 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.848599    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fd5deac-72f2-4b25-9db8-82735b3ad959-cert\") pod \"kserve-controller-manager-d6f4b65b8-gmncm\" (UID: \"6fd5deac-72f2-4b25-9db8-82735b3ad959\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gmncm"
Sep 14 13:05:43.850826 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.850808    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fd5deac-72f2-4b25-9db8-82735b3ad959-cert\") pod \"kserve-controller-manager-d6f4b65b8-gmncm\" (UID: \"6fd5deac-72f2-4b25-9db8-82735b3ad959\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gmncm"
Sep 14 13:05:43.866477 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:43.866445    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpq4w" event={"ID":"120fe86c-4cf8-488d-bdb3-0c5a120d35dd","Type":"ContainerStarted","Data":"e5e49aeb458388f120ac3d33a837111073e307d79bb782ef9bbf3e7288a70c7c"}
Sep 14 13:05:44.080303 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:44.080245    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gmncm"
Sep 14 13:05:44.361943 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:44.361918    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gmncm"]
Sep 14 13:05:44.421625 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:05:44.421596    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd5deac_72f2_4b25_9db8_82735b3ad959.slice/crio-336637a0702b9c4fc59f90d1bdc983cb0349a901b865edf65a7622c741780c99 WatchSource:0}: Error finding container 336637a0702b9c4fc59f90d1bdc983cb0349a901b865edf65a7622c741780c99: Status 404 returned error can't find the container with id 336637a0702b9c4fc59f90d1bdc983cb0349a901b865edf65a7622c741780c99
Sep 14 13:05:44.872921 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:44.872884    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gmncm" event={"ID":"6fd5deac-72f2-4b25-9db8-82735b3ad959","Type":"ContainerStarted","Data":"336637a0702b9c4fc59f90d1bdc983cb0349a901b865edf65a7622c741780c99"}
Sep 14 13:05:45.013151 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.013115    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"]
Sep 14 13:05:45.013376 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.013348    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p" podUID="864604c0-07c8-4a82-8b5f-092e5d70457c" containerName="discovery" containerID="cri-o://f134ba2cfee025dcec52c47f8d85b4fa2219d1d1b6d340d670ac842db113fb1b" gracePeriod=30
Sep 14 13:05:45.277190 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.277166    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:45.361759 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.361708    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-csr-ca-configmap\") pod \"864604c0-07c8-4a82-8b5f-092e5d70457c\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") "
Sep 14 13:05:45.361908 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.361777    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/864604c0-07c8-4a82-8b5f-092e5d70457c-cacerts\") pod \"864604c0-07c8-4a82-8b5f-092e5d70457c\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") "
Sep 14 13:05:45.361908 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.361822    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/864604c0-07c8-4a82-8b5f-092e5d70457c-local-certs\") pod \"864604c0-07c8-4a82-8b5f-092e5d70457c\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") "
Sep 14 13:05:45.361908 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.361847    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-csr-dns-cert\") pod \"864604c0-07c8-4a82-8b5f-092e5d70457c\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") "
Sep 14 13:05:45.361908 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.361882    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9tnz\" (UniqueName: \"kubernetes.io/projected/864604c0-07c8-4a82-8b5f-092e5d70457c-kube-api-access-n9tnz\") pod \"864604c0-07c8-4a82-8b5f-092e5d70457c\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") "
Sep 14 13:05:45.362451 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.361955    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-kubeconfig\") pod \"864604c0-07c8-4a82-8b5f-092e5d70457c\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") "
Sep 14 13:05:45.362451 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.362010    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-token\") pod \"864604c0-07c8-4a82-8b5f-092e5d70457c\" (UID: \"864604c0-07c8-4a82-8b5f-092e5d70457c\") "
Sep 14 13:05:45.362451 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.362130    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "864604c0-07c8-4a82-8b5f-092e5d70457c" (UID: "864604c0-07c8-4a82-8b5f-092e5d70457c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:05:45.362451 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.362221    2557 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-csr-ca-configmap\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:05:45.364703 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.364573    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-token" (OuterVolumeSpecName: "istio-token") pod "864604c0-07c8-4a82-8b5f-092e5d70457c" (UID: "864604c0-07c8-4a82-8b5f-092e5d70457c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:05:45.364703 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.364672    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864604c0-07c8-4a82-8b5f-092e5d70457c-cacerts" (OuterVolumeSpecName: "cacerts") pod "864604c0-07c8-4a82-8b5f-092e5d70457c" (UID: "864604c0-07c8-4a82-8b5f-092e5d70457c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:05:45.365196 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.365088    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "864604c0-07c8-4a82-8b5f-092e5d70457c" (UID: "864604c0-07c8-4a82-8b5f-092e5d70457c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:05:45.365196 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.365167    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "864604c0-07c8-4a82-8b5f-092e5d70457c" (UID: "864604c0-07c8-4a82-8b5f-092e5d70457c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:05:45.365582 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.365539    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864604c0-07c8-4a82-8b5f-092e5d70457c-local-certs" (OuterVolumeSpecName: "local-certs") pod "864604c0-07c8-4a82-8b5f-092e5d70457c" (UID: "864604c0-07c8-4a82-8b5f-092e5d70457c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:05:45.366006 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.365976    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864604c0-07c8-4a82-8b5f-092e5d70457c-kube-api-access-n9tnz" (OuterVolumeSpecName: "kube-api-access-n9tnz") pod "864604c0-07c8-4a82-8b5f-092e5d70457c" (UID: "864604c0-07c8-4a82-8b5f-092e5d70457c"). InnerVolumeSpecName "kube-api-access-n9tnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:05:45.462840 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.462813    2557 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/864604c0-07c8-4a82-8b5f-092e5d70457c-cacerts\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:05:45.462982 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.462844    2557 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/864604c0-07c8-4a82-8b5f-092e5d70457c-local-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:05:45.462982 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.462860    2557 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-csr-dns-cert\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:05:45.462982 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.462873    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9tnz\" (UniqueName: \"kubernetes.io/projected/864604c0-07c8-4a82-8b5f-092e5d70457c-kube-api-access-n9tnz\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:05:45.462982 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.462886    2557 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-kubeconfig\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:05:45.462982 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.462899    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/864604c0-07c8-4a82-8b5f-092e5d70457c-istio-token\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:05:45.879091 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.878995    2557 generic.go:356] "Generic (PLEG): container finished" podID="864604c0-07c8-4a82-8b5f-092e5d70457c" containerID="f134ba2cfee025dcec52c47f8d85b4fa2219d1d1b6d340d670ac842db113fb1b" exitCode=0
Sep 14 13:05:45.879091 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.879055    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"
Sep 14 13:05:45.879091 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.879080    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p" event={"ID":"864604c0-07c8-4a82-8b5f-092e5d70457c","Type":"ContainerDied","Data":"f134ba2cfee025dcec52c47f8d85b4fa2219d1d1b6d340d670ac842db113fb1b"}
Sep 14 13:05:45.879347 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.879166    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p" event={"ID":"864604c0-07c8-4a82-8b5f-092e5d70457c","Type":"ContainerDied","Data":"75c10f3b0b475ab1b8ee68f405be4efd62b293181e20214a55aa67dcc004ddf9"}
Sep 14 13:05:45.879347 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.879189    2557 scope.go:117] "RemoveContainer" containerID="f134ba2cfee025dcec52c47f8d85b4fa2219d1d1b6d340d670ac842db113fb1b"
Sep 14 13:05:45.905051 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.905027    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"]
Sep 14 13:05:45.913888 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:45.913865    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6f28p"]
Sep 14 13:05:46.048692 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:46.048661    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="864604c0-07c8-4a82-8b5f-092e5d70457c" path="/var/lib/kubelet/pods/864604c0-07c8-4a82-8b5f-092e5d70457c/volumes"
Sep 14 13:05:46.653750 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:46.653530    2557 scope.go:117] "RemoveContainer" containerID="f134ba2cfee025dcec52c47f8d85b4fa2219d1d1b6d340d670ac842db113fb1b"
Sep 14 13:05:46.653999 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:05:46.653898    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f134ba2cfee025dcec52c47f8d85b4fa2219d1d1b6d340d670ac842db113fb1b\": container with ID starting with f134ba2cfee025dcec52c47f8d85b4fa2219d1d1b6d340d670ac842db113fb1b not found: ID does not exist" containerID="f134ba2cfee025dcec52c47f8d85b4fa2219d1d1b6d340d670ac842db113fb1b"
Sep 14 13:05:46.653999 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:46.653931    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f134ba2cfee025dcec52c47f8d85b4fa2219d1d1b6d340d670ac842db113fb1b"} err="failed to get container status \"f134ba2cfee025dcec52c47f8d85b4fa2219d1d1b6d340d670ac842db113fb1b\": rpc error: code = NotFound desc = could not find container \"f134ba2cfee025dcec52c47f8d85b4fa2219d1d1b6d340d670ac842db113fb1b\": container with ID starting with f134ba2cfee025dcec52c47f8d85b4fa2219d1d1b6d340d670ac842db113fb1b not found: ID does not exist"
Sep 14 13:05:46.884753 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:46.884711    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpq4w" event={"ID":"120fe86c-4cf8-488d-bdb3-0c5a120d35dd","Type":"ContainerStarted","Data":"508e27a104bb315c9ec8829bb0a8e5acd65d626d0a78a262de3c2086d91b2eaf"}
Sep 14 13:05:46.884895 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:46.884880    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpq4w"
Sep 14 13:05:46.886316 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:46.886294    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gmncm" event={"ID":"6fd5deac-72f2-4b25-9db8-82735b3ad959","Type":"ContainerStarted","Data":"62181b74425d6452d2f81aa4bb74a65dbcb3f286fbf6ea220df78dcd66df5b0a"}
Sep 14 13:05:46.886444 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:46.886432    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gmncm"
Sep 14 13:05:46.905774 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:46.905673    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpq4w" podStartSLOduration=1.005884607 podStartE2EDuration="3.905660938s" totalImagesPullingTime="2.899776331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:05:46.90359481 +0000 UTC m=+465.338270329" watchObservedRunningTime="2026-09-14 13:05:46.905660938 +0000 UTC m=+465.340336451"
Sep 14 13:05:46.924242 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:46.924201    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gmncm" podStartSLOduration=1.6815960030000001 podStartE2EDuration="3.924189506s" totalImagesPullingTime="2.242593503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:05:46.922800575 +0000 UTC m=+465.357476093" watchObservedRunningTime="2026-09-14 13:05:46.924189506 +0000 UTC m=+465.358865024"
Sep 14 13:05:48.852247 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:05:48.852220    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-xfqw6"
Sep 14 13:06:17.893620 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:17.893579    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpq4w"
Sep 14 13:06:17.896538 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:17.896508    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gmncm"
Sep 14 13:06:50.805793 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:50.805751    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"]
Sep 14 13:06:50.806324 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:50.806057    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="864604c0-07c8-4a82-8b5f-092e5d70457c" containerName="discovery"
Sep 14 13:06:50.806324 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:50.806068    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="864604c0-07c8-4a82-8b5f-092e5d70457c" containerName="discovery"
Sep 14 13:06:50.806324 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:50.806112    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="864604c0-07c8-4a82-8b5f-092e5d70457c" containerName="discovery"
Sep 14 13:06:50.809008 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:50.808990    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"
Sep 14 13:06:50.811942 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:50.811915    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 14 13:06:50.812062 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:50.811961    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 14 13:06:50.812062 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:50.812052    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fbs72\""
Sep 14 13:06:50.822080 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:50.822055    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"]
Sep 14 13:06:50.901083 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:50.901058    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a977a168-10fc-42f2-81dc-b91ba01edee2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v82zf\" (UID: \"a977a168-10fc-42f2-81dc-b91ba01edee2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"
Sep 14 13:06:50.901201 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:50.901121    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn9d5\" (UniqueName: \"kubernetes.io/projected/a977a168-10fc-42f2-81dc-b91ba01edee2-kube-api-access-pn9d5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v82zf\" (UID: \"a977a168-10fc-42f2-81dc-b91ba01edee2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"
Sep 14 13:06:51.002223 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:51.002189    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a977a168-10fc-42f2-81dc-b91ba01edee2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v82zf\" (UID: \"a977a168-10fc-42f2-81dc-b91ba01edee2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"
Sep 14 13:06:51.002371 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:51.002249    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn9d5\" (UniqueName: \"kubernetes.io/projected/a977a168-10fc-42f2-81dc-b91ba01edee2-kube-api-access-pn9d5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v82zf\" (UID: \"a977a168-10fc-42f2-81dc-b91ba01edee2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"
Sep 14 13:06:51.002579 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:51.002558    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a977a168-10fc-42f2-81dc-b91ba01edee2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v82zf\" (UID: \"a977a168-10fc-42f2-81dc-b91ba01edee2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"
Sep 14 13:06:51.010950 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:51.010919    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn9d5\" (UniqueName: \"kubernetes.io/projected/a977a168-10fc-42f2-81dc-b91ba01edee2-kube-api-access-pn9d5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v82zf\" (UID: \"a977a168-10fc-42f2-81dc-b91ba01edee2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"
Sep 14 13:06:51.118379 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:51.118307    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"
Sep 14 13:06:51.394893 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:51.394866    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"]
Sep 14 13:06:51.397590 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:06:51.397560    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda977a168_10fc_42f2_81dc_b91ba01edee2.slice/crio-2b15347a5b264b43628c5a6b451670f9e50974eccd9a37a0d2863a4d2a780526 WatchSource:0}: Error finding container 2b15347a5b264b43628c5a6b451670f9e50974eccd9a37a0d2863a4d2a780526: Status 404 returned error can't find the container with id 2b15347a5b264b43628c5a6b451670f9e50974eccd9a37a0d2863a4d2a780526
Sep 14 13:06:52.126938 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:52.126864    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf" event={"ID":"a977a168-10fc-42f2-81dc-b91ba01edee2","Type":"ContainerStarted","Data":"2b15347a5b264b43628c5a6b451670f9e50974eccd9a37a0d2863a4d2a780526"}
Sep 14 13:06:52.700992 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:52.700904    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4874m"]
Sep 14 13:06:52.706035 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:52.706015    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4874m"
Sep 14 13:06:52.708897 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:52.708874    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 14 13:06:52.709006 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:52.708874    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-6628r\""
Sep 14 13:06:52.712507 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:52.712475    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ctg4\" (UniqueName: \"kubernetes.io/projected/30ca4937-6ff6-411a-8144-1d6c6733269d-kube-api-access-9ctg4\") pod \"dns-operator-controller-manager-7b84cd7c9c-4874m\" (UID: \"30ca4937-6ff6-411a-8144-1d6c6733269d\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4874m"
Sep 14 13:06:52.716799 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:52.716726    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4874m"]
Sep 14 13:06:52.813497 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:52.813467    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ctg4\" (UniqueName: \"kubernetes.io/projected/30ca4937-6ff6-411a-8144-1d6c6733269d-kube-api-access-9ctg4\") pod \"dns-operator-controller-manager-7b84cd7c9c-4874m\" (UID: \"30ca4937-6ff6-411a-8144-1d6c6733269d\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4874m"
Sep 14 13:06:52.827471 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:52.827432    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ctg4\" (UniqueName: \"kubernetes.io/projected/30ca4937-6ff6-411a-8144-1d6c6733269d-kube-api-access-9ctg4\") pod \"dns-operator-controller-manager-7b84cd7c9c-4874m\" (UID: \"30ca4937-6ff6-411a-8144-1d6c6733269d\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4874m"
Sep 14 13:06:53.020001 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:53.019917    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4874m"
Sep 14 13:06:53.725088 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:53.725064    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4874m"]
Sep 14 13:06:53.728020 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:06:53.727990    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ca4937_6ff6_411a_8144_1d6c6733269d.slice/crio-8e287ce507ac57f42aae1ff50741e6fe451bb81a5ce1c0cb9c5b9f599e4d41c4 WatchSource:0}: Error finding container 8e287ce507ac57f42aae1ff50741e6fe451bb81a5ce1c0cb9c5b9f599e4d41c4: Status 404 returned error can't find the container with id 8e287ce507ac57f42aae1ff50741e6fe451bb81a5ce1c0cb9c5b9f599e4d41c4
Sep 14 13:06:54.136830 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:54.136753    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4874m" event={"ID":"30ca4937-6ff6-411a-8144-1d6c6733269d","Type":"ContainerStarted","Data":"8e287ce507ac57f42aae1ff50741e6fe451bb81a5ce1c0cb9c5b9f599e4d41c4"}
Sep 14 13:06:55.102547 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:55.102472    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2"]
Sep 14 13:06:55.105563 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:55.105548    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2"
Sep 14 13:06:55.108336 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:55.108313    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-cn2zr\""
Sep 14 13:06:55.117134 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:55.117114    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2"]
Sep 14 13:06:55.128644 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:55.128622    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9qmn\" (UniqueName: \"kubernetes.io/projected/0a75d8df-dd29-4758-9bfb-f3d233535496-kube-api-access-f9qmn\") pod \"limitador-operator-controller-manager-74495c69f8-hrqw2\" (UID: \"0a75d8df-dd29-4758-9bfb-f3d233535496\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2"
Sep 14 13:06:55.229352 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:55.229319    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9qmn\" (UniqueName: \"kubernetes.io/projected/0a75d8df-dd29-4758-9bfb-f3d233535496-kube-api-access-f9qmn\") pod \"limitador-operator-controller-manager-74495c69f8-hrqw2\" (UID: \"0a75d8df-dd29-4758-9bfb-f3d233535496\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2"
Sep 14 13:06:55.238725 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:55.238699    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9qmn\" (UniqueName: \"kubernetes.io/projected/0a75d8df-dd29-4758-9bfb-f3d233535496-kube-api-access-f9qmn\") pod \"limitador-operator-controller-manager-74495c69f8-hrqw2\" (UID: \"0a75d8df-dd29-4758-9bfb-f3d233535496\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2"
Sep 14 13:06:55.416449 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:55.416361    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2"
Sep 14 13:06:55.695424 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:55.695399    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2"]
Sep 14 13:06:56.018717 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:06:56.018646    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a75d8df_dd29_4758_9bfb_f3d233535496.slice/crio-39100a4b7d7dc2a7a74c149de4f3f6c68263fbf26648f5ad8dd01f12833e7ee6 WatchSource:0}: Error finding container 39100a4b7d7dc2a7a74c149de4f3f6c68263fbf26648f5ad8dd01f12833e7ee6: Status 404 returned error can't find the container with id 39100a4b7d7dc2a7a74c149de4f3f6c68263fbf26648f5ad8dd01f12833e7ee6
Sep 14 13:06:56.146520 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:56.146489    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf" event={"ID":"a977a168-10fc-42f2-81dc-b91ba01edee2","Type":"ContainerStarted","Data":"c5949b011e8f9b4fc46744d623908996e481383fa33edbf275d81711cca25cef"}
Sep 14 13:06:56.146890 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:56.146570    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"
Sep 14 13:06:56.151427 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:56.151394    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4874m" event={"ID":"30ca4937-6ff6-411a-8144-1d6c6733269d","Type":"ContainerStarted","Data":"d144b38a1e051e3194e40fa487824688ce2efbcbb89cf8f3f25f1902a85e008b"}
Sep 14 13:06:56.151557 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:56.151474    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4874m"
Sep 14 13:06:56.153443 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:56.153419    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2" event={"ID":"0a75d8df-dd29-4758-9bfb-f3d233535496","Type":"ContainerStarted","Data":"39100a4b7d7dc2a7a74c149de4f3f6c68263fbf26648f5ad8dd01f12833e7ee6"}
Sep 14 13:06:56.176682 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:56.176635    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf" podStartSLOduration=2.458655648 podStartE2EDuration="6.176621939s" totalImagesPullingTime="3.717966291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:06:56.174483478 +0000 UTC m=+534.609158997" watchObservedRunningTime="2026-09-14 13:06:56.176621939 +0000 UTC m=+534.611297457"
Sep 14 13:06:56.201092 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:56.201030    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4874m" podStartSLOduration=1.851940542 podStartE2EDuration="4.201015271s" totalImagesPullingTime="2.349074729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:06:56.200196447 +0000 UTC m=+534.634871966" watchObservedRunningTime="2026-09-14 13:06:56.201015271 +0000 UTC m=+534.635690789"
Sep 14 13:06:57.824519 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:57.824443    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-9q2vl"]
Sep 14 13:06:57.827642 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:57.827625    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-9q2vl"
Sep 14 13:06:57.830235 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:57.830216    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-nbnvq\""
Sep 14 13:06:57.839547 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:57.839524    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-9q2vl"]
Sep 14 13:06:57.846121 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:57.846098    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpl42\" (UniqueName: \"kubernetes.io/projected/49fac7d8-720a-4482-a3bb-fec0ac32d1a4-kube-api-access-jpl42\") pod \"authorino-operator-76b478bd78-9q2vl\" (UID: \"49fac7d8-720a-4482-a3bb-fec0ac32d1a4\") " pod="kuadrant-system/authorino-operator-76b478bd78-9q2vl"
Sep 14 13:06:57.946912 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:57.946877    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpl42\" (UniqueName: \"kubernetes.io/projected/49fac7d8-720a-4482-a3bb-fec0ac32d1a4-kube-api-access-jpl42\") pod \"authorino-operator-76b478bd78-9q2vl\" (UID: \"49fac7d8-720a-4482-a3bb-fec0ac32d1a4\") " pod="kuadrant-system/authorino-operator-76b478bd78-9q2vl"
Sep 14 13:06:57.958833 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:57.958801    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpl42\" (UniqueName: \"kubernetes.io/projected/49fac7d8-720a-4482-a3bb-fec0ac32d1a4-kube-api-access-jpl42\") pod \"authorino-operator-76b478bd78-9q2vl\" (UID: \"49fac7d8-720a-4482-a3bb-fec0ac32d1a4\") " pod="kuadrant-system/authorino-operator-76b478bd78-9q2vl"
Sep 14 13:06:58.137654 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:58.137564    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-9q2vl"
Sep 14 13:06:58.166166 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:58.166135    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2" event={"ID":"0a75d8df-dd29-4758-9bfb-f3d233535496","Type":"ContainerStarted","Data":"1ce39db2c6699461268afdb2f4913da2415c8e9472b0da75bf9e2f31539aec4b"}
Sep 14 13:06:58.166311 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:58.166208    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2"
Sep 14 13:06:58.187159 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:58.187098    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2" podStartSLOduration=1.688676553 podStartE2EDuration="3.187075596s" totalImagesPullingTime="1.498399043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:06:58.184436109 +0000 UTC m=+536.619111629" watchObservedRunningTime="2026-09-14 13:06:58.187075596 +0000 UTC m=+536.621751116"
Sep 14 13:06:58.229046 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:58.229014    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-kv2b4"]
Sep 14 13:06:58.232282 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:58.232265    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kv2b4"
Sep 14 13:06:58.235201 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:58.235180    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 14 13:06:58.235201 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:58.235190    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 14 13:06:58.235337 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:58.235206    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bznfl\""
Sep 14 13:06:58.245363 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:58.243506    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-kv2b4"]
Sep 14 13:06:58.248193 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:58.248172    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b8271e98-9947-44ab-92b9-278948b9b3ec-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-kv2b4\" (UID: \"b8271e98-9947-44ab-92b9-278948b9b3ec\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kv2b4"
Sep 14 13:06:58.248293 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:58.248211    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8271e98-9947-44ab-92b9-278948b9b3ec-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-kv2b4\" (UID: \"b8271e98-9947-44ab-92b9-278948b9b3ec\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kv2b4"
Sep 14 13:06:58.248293 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:58.248266    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppmpt\" (UniqueName: \"kubernetes.io/projected/b8271e98-9947-44ab-92b9-278948b9b3ec-kube-api-access-ppmpt\") pod \"kuadrant-console-plugin-fbdd4f445-kv2b4\" (UID: \"b8271e98-9947-44ab-92b9-278948b9b3ec\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kv2b4"
Sep 14 13:06:58.349169 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:58.349131    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8271e98-9947-44ab-92b9-278948b9b3ec-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-kv2b4\" (UID: \"b8271e98-9947-44ab-92b9-278948b9b3ec\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kv2b4"
Sep 14 13:06:58.349355 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:58.349196    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppmpt\" (UniqueName: \"kubernetes.io/projected/b8271e98-9947-44ab-92b9-278948b9b3ec-kube-api-access-ppmpt\") pod \"kuadrant-console-plugin-fbdd4f445-kv2b4\" (UID: \"b8271e98-9947-44ab-92b9-278948b9b3ec\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kv2b4"
Sep 14 13:06:58.349355 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:58.349247    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b8271e98-9947-44ab-92b9-278948b9b3ec-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-kv2b4\" (UID: \"b8271e98-9947-44ab-92b9-278948b9b3ec\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kv2b4"
Sep 14 13:06:58.349355 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:06:58.349272    2557 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 14 13:06:58.349355 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:06:58.349342    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8271e98-9947-44ab-92b9-278948b9b3ec-plugin-serving-cert podName:b8271e98-9947-44ab-92b9-278948b9b3ec nodeName:}" failed. No retries permitted until 2026-09-14 13:06:58.849324335 +0000 UTC m=+537.283999837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/b8271e98-9947-44ab-92b9-278948b9b3ec-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-kv2b4" (UID: "b8271e98-9947-44ab-92b9-278948b9b3ec") : secret "plugin-serving-cert" not found
Sep 14 13:06:58.349955 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:58.349933    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b8271e98-9947-44ab-92b9-278948b9b3ec-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-kv2b4\" (UID: \"b8271e98-9947-44ab-92b9-278948b9b3ec\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kv2b4"
Sep 14 13:06:58.362386 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:58.362362    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppmpt\" (UniqueName: \"kubernetes.io/projected/b8271e98-9947-44ab-92b9-278948b9b3ec-kube-api-access-ppmpt\") pod \"kuadrant-console-plugin-fbdd4f445-kv2b4\" (UID: \"b8271e98-9947-44ab-92b9-278948b9b3ec\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kv2b4"
Sep 14 13:06:58.417814 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:58.417660    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-9q2vl"]
Sep 14 13:06:58.420392 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:06:58.420362    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49fac7d8_720a_4482_a3bb_fec0ac32d1a4.slice/crio-6520d01b3e9afaf78e49132a9d3181168f50612ef59cdeaed54c00ac2bc8ef25 WatchSource:0}: Error finding container 6520d01b3e9afaf78e49132a9d3181168f50612ef59cdeaed54c00ac2bc8ef25: Status 404 returned error can't find the container with id 6520d01b3e9afaf78e49132a9d3181168f50612ef59cdeaed54c00ac2bc8ef25
Sep 14 13:06:58.851565 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:58.851522    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8271e98-9947-44ab-92b9-278948b9b3ec-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-kv2b4\" (UID: \"b8271e98-9947-44ab-92b9-278948b9b3ec\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kv2b4"
Sep 14 13:06:58.854034 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:58.854012    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8271e98-9947-44ab-92b9-278948b9b3ec-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-kv2b4\" (UID: \"b8271e98-9947-44ab-92b9-278948b9b3ec\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kv2b4"
Sep 14 13:06:59.143683 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:59.143610    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kv2b4"
Sep 14 13:06:59.173172 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:59.173112    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-9q2vl" event={"ID":"49fac7d8-720a-4482-a3bb-fec0ac32d1a4","Type":"ContainerStarted","Data":"6520d01b3e9afaf78e49132a9d3181168f50612ef59cdeaed54c00ac2bc8ef25"}
Sep 14 13:06:59.427922 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:06:59.427822    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-kv2b4"]
Sep 14 13:06:59.594865 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:06:59.594826    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8271e98_9947_44ab_92b9_278948b9b3ec.slice/crio-ff218a01346c06198087b25ac17bf7b42df72988a5f65b2cd4cb96d3d1375e11 WatchSource:0}: Error finding container ff218a01346c06198087b25ac17bf7b42df72988a5f65b2cd4cb96d3d1375e11: Status 404 returned error can't find the container with id ff218a01346c06198087b25ac17bf7b42df72988a5f65b2cd4cb96d3d1375e11
Sep 14 13:07:00.180467 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:00.180433    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-9q2vl" event={"ID":"49fac7d8-720a-4482-a3bb-fec0ac32d1a4","Type":"ContainerStarted","Data":"928c6e47e40a92aa0c2d7e1adde174cd085282969257811ef5e8417e535c665f"}
Sep 14 13:07:00.180974 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:00.180520    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-9q2vl"
Sep 14 13:07:00.181784 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:00.181761    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kv2b4" event={"ID":"b8271e98-9947-44ab-92b9-278948b9b3ec","Type":"ContainerStarted","Data":"ff218a01346c06198087b25ac17bf7b42df72988a5f65b2cd4cb96d3d1375e11"}
Sep 14 13:07:00.200354 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:00.200303    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-9q2vl" podStartSLOduration=1.972849568 podStartE2EDuration="3.200285651s" totalImagesPullingTime="1.227436083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:06:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:07:00.198349177 +0000 UTC m=+538.633024697" watchObservedRunningTime="2026-09-14 13:07:00.200285651 +0000 UTC m=+538.634961170"
Sep 14 13:07:05.206848 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:05.206806    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kv2b4" event={"ID":"b8271e98-9947-44ab-92b9-278948b9b3ec","Type":"ContainerStarted","Data":"cfe10b5a45d85bf75fc0f93450aa3dbbb13cbd57e1ae21a82c2a4644bb8f420b"}
Sep 14 13:07:05.224442 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:05.224395    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kv2b4" podStartSLOduration=2.354476119 podStartE2EDuration="7.224381982s" totalImagesPullingTime="4.869905863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:06:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:07:05.223067029 +0000 UTC m=+543.657742547" watchObservedRunningTime="2026-09-14 13:07:05.224381982 +0000 UTC m=+543.659057501"
Sep 14 13:07:07.161010 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:07.160981    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"
Sep 14 13:07:07.161371 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:07.161051    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4874m"
Sep 14 13:07:09.175786 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.175730    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2"
Sep 14 13:07:09.365197 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.365162    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"]
Sep 14 13:07:09.365366 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.365343    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf" podUID="a977a168-10fc-42f2-81dc-b91ba01edee2" containerName="manager" containerID="cri-o://c5949b011e8f9b4fc46744d623908996e481383fa33edbf275d81711cca25cef" gracePeriod=2
Sep 14 13:07:09.380919 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.380754    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"]
Sep 14 13:07:09.388927 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:07:09.388885    2557 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"a977a168-10fc-42f2-81dc-b91ba01edee2\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-v82zf\": pods \"kuadrant-operator-controller-manager-76f8f686bf-v82zf\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"
Sep 14 13:07:09.392703 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.392666    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2"]
Sep 14 13:07:09.392887 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.392861    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2" podUID="0a75d8df-dd29-4758-9bfb-f3d233535496" containerName="manager" containerID="cri-o://1ce39db2c6699461268afdb2f4913da2415c8e9472b0da75bf9e2f31539aec4b" gracePeriod=2
Sep 14 13:07:09.396678 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.396653    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz"]
Sep 14 13:07:09.397042 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.397026    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a977a168-10fc-42f2-81dc-b91ba01edee2" containerName="manager"
Sep 14 13:07:09.397129 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.397044    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a977a168-10fc-42f2-81dc-b91ba01edee2" containerName="manager"
Sep 14 13:07:09.397184 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.397132    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a977a168-10fc-42f2-81dc-b91ba01edee2" containerName="manager"
Sep 14 13:07:09.400082 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.400063    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz"
Sep 14 13:07:09.406267 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.406245    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2"]
Sep 14 13:07:09.414373 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:07:09.414329    2557 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0a75d8df-dd29-4758-9bfb-f3d233535496\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-hrqw2\": pods \"limitador-operator-controller-manager-74495c69f8-hrqw2\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2"
Sep 14 13:07:09.418518 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:07:09.418484    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-v82zf\" is forbidden: User \"system:node:ip-10-0-129-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-61.ec2.internal' and this object" podUID="a977a168-10fc-42f2-81dc-b91ba01edee2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"
Sep 14 13:07:09.420770 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.420470    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz"]
Sep 14 13:07:09.420882 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.420819    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86f0158c-fb62-4d2f-af5e-07dc2eb98205-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g2dlz\" (UID: \"86f0158c-fb62-4d2f-af5e-07dc2eb98205\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz"
Sep 14 13:07:09.420882 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.420876    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brmxw\" (UniqueName: \"kubernetes.io/projected/86f0158c-fb62-4d2f-af5e-07dc2eb98205-kube-api-access-brmxw\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g2dlz\" (UID: \"86f0158c-fb62-4d2f-af5e-07dc2eb98205\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz"
Sep 14 13:07:09.423963 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.423943    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-xgkvh"]
Sep 14 13:07:09.424257 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.424241    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a75d8df-dd29-4758-9bfb-f3d233535496" containerName="manager"
Sep 14 13:07:09.424320 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.424261    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a75d8df-dd29-4758-9bfb-f3d233535496" containerName="manager"
Sep 14 13:07:09.424373 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.424352    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a75d8df-dd29-4758-9bfb-f3d233535496" containerName="manager"
Sep 14 13:07:09.427440 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.427384    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xgkvh"
Sep 14 13:07:09.435276 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.435254    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-xgkvh"]
Sep 14 13:07:09.445594 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:07:09.445563    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-v82zf\" is forbidden: User \"system:node:ip-10-0-129-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-61.ec2.internal' and this object" podUID="a977a168-10fc-42f2-81dc-b91ba01edee2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"
Sep 14 13:07:09.447749 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:07:09.447710    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-hrqw2\" is forbidden: User \"system:node:ip-10-0-129-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-61.ec2.internal' and this object" podUID="0a75d8df-dd29-4758-9bfb-f3d233535496" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2"
Sep 14 13:07:09.469311 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:07:09.469271    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-v82zf\" is forbidden: User \"system:node:ip-10-0-129-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-61.ec2.internal' and this object" podUID="a977a168-10fc-42f2-81dc-b91ba01edee2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"
Sep 14 13:07:09.471421 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:07:09.471388    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-hrqw2\" is forbidden: User \"system:node:ip-10-0-129-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-61.ec2.internal' and this object" podUID="0a75d8df-dd29-4758-9bfb-f3d233535496" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2"
Sep 14 13:07:09.521493 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.521444    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86f0158c-fb62-4d2f-af5e-07dc2eb98205-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g2dlz\" (UID: \"86f0158c-fb62-4d2f-af5e-07dc2eb98205\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz"
Sep 14 13:07:09.521700 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.521619    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brmxw\" (UniqueName: \"kubernetes.io/projected/86f0158c-fb62-4d2f-af5e-07dc2eb98205-kube-api-access-brmxw\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g2dlz\" (UID: \"86f0158c-fb62-4d2f-af5e-07dc2eb98205\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz"
Sep 14 13:07:09.521828 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.521809    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86f0158c-fb62-4d2f-af5e-07dc2eb98205-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g2dlz\" (UID: \"86f0158c-fb62-4d2f-af5e-07dc2eb98205\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz"
Sep 14 13:07:09.535120 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.535065    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brmxw\" (UniqueName: \"kubernetes.io/projected/86f0158c-fb62-4d2f-af5e-07dc2eb98205-kube-api-access-brmxw\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g2dlz\" (UID: \"86f0158c-fb62-4d2f-af5e-07dc2eb98205\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz"
Sep 14 13:07:09.619907 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.619885    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"
Sep 14 13:07:09.622000 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.621978    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a977a168-10fc-42f2-81dc-b91ba01edee2-extensions-socket-volume\") pod \"a977a168-10fc-42f2-81dc-b91ba01edee2\" (UID: \"a977a168-10fc-42f2-81dc-b91ba01edee2\") "
Sep 14 13:07:09.622113 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.622032    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn9d5\" (UniqueName: \"kubernetes.io/projected/a977a168-10fc-42f2-81dc-b91ba01edee2-kube-api-access-pn9d5\") pod \"a977a168-10fc-42f2-81dc-b91ba01edee2\" (UID: \"a977a168-10fc-42f2-81dc-b91ba01edee2\") "
Sep 14 13:07:09.622204 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.622171    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbrc6\" (UniqueName: \"kubernetes.io/projected/27b02f0d-9f27-4834-b312-ddc117f4433b-kube-api-access-fbrc6\") pod \"limitador-operator-controller-manager-74495c69f8-xgkvh\" (UID: \"27b02f0d-9f27-4834-b312-ddc117f4433b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xgkvh"
Sep 14 13:07:09.622348 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.622320    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a977a168-10fc-42f2-81dc-b91ba01edee2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a977a168-10fc-42f2-81dc-b91ba01edee2" (UID: "a977a168-10fc-42f2-81dc-b91ba01edee2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:07:09.622638 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:07:09.622613    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-v82zf\" is forbidden: User \"system:node:ip-10-0-129-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-61.ec2.internal' and this object" podUID="a977a168-10fc-42f2-81dc-b91ba01edee2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"
Sep 14 13:07:09.624132 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.624111    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a977a168-10fc-42f2-81dc-b91ba01edee2-kube-api-access-pn9d5" (OuterVolumeSpecName: "kube-api-access-pn9d5") pod "a977a168-10fc-42f2-81dc-b91ba01edee2" (UID: "a977a168-10fc-42f2-81dc-b91ba01edee2"). InnerVolumeSpecName "kube-api-access-pn9d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:07:09.624544 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.624527    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2"
Sep 14 13:07:09.625250 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:07:09.625223    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-hrqw2\" is forbidden: User \"system:node:ip-10-0-129-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-61.ec2.internal' and this object" podUID="0a75d8df-dd29-4758-9bfb-f3d233535496" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2"
Sep 14 13:07:09.627819 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:07:09.627797    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-v82zf\" is forbidden: User \"system:node:ip-10-0-129-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-61.ec2.internal' and this object" podUID="a977a168-10fc-42f2-81dc-b91ba01edee2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"
Sep 14 13:07:09.630108 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:07:09.630087    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-hrqw2\" is forbidden: User \"system:node:ip-10-0-129-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-61.ec2.internal' and this object" podUID="0a75d8df-dd29-4758-9bfb-f3d233535496" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2"
Sep 14 13:07:09.722833 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.722800    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9qmn\" (UniqueName: \"kubernetes.io/projected/0a75d8df-dd29-4758-9bfb-f3d233535496-kube-api-access-f9qmn\") pod \"0a75d8df-dd29-4758-9bfb-f3d233535496\" (UID: \"0a75d8df-dd29-4758-9bfb-f3d233535496\") "
Sep 14 13:07:09.722994 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.722931    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbrc6\" (UniqueName: \"kubernetes.io/projected/27b02f0d-9f27-4834-b312-ddc117f4433b-kube-api-access-fbrc6\") pod \"limitador-operator-controller-manager-74495c69f8-xgkvh\" (UID: \"27b02f0d-9f27-4834-b312-ddc117f4433b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xgkvh"
Sep 14 13:07:09.722994 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.722982    2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a977a168-10fc-42f2-81dc-b91ba01edee2-extensions-socket-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:07:09.722994 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.722992    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn9d5\" (UniqueName: \"kubernetes.io/projected/a977a168-10fc-42f2-81dc-b91ba01edee2-kube-api-access-pn9d5\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:07:09.724876 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.724853    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a75d8df-dd29-4758-9bfb-f3d233535496-kube-api-access-f9qmn" (OuterVolumeSpecName: "kube-api-access-f9qmn") pod "0a75d8df-dd29-4758-9bfb-f3d233535496" (UID: "0a75d8df-dd29-4758-9bfb-f3d233535496"). InnerVolumeSpecName "kube-api-access-f9qmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:07:09.734614 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.734596    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbrc6\" (UniqueName: \"kubernetes.io/projected/27b02f0d-9f27-4834-b312-ddc117f4433b-kube-api-access-fbrc6\") pod \"limitador-operator-controller-manager-74495c69f8-xgkvh\" (UID: \"27b02f0d-9f27-4834-b312-ddc117f4433b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xgkvh"
Sep 14 13:07:09.795375 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.795350    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz"
Sep 14 13:07:09.802443 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.802420    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xgkvh"
Sep 14 13:07:09.823523 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:09.823490    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9qmn\" (UniqueName: \"kubernetes.io/projected/0a75d8df-dd29-4758-9bfb-f3d233535496-kube-api-access-f9qmn\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:07:10.048030 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:10.047952    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a75d8df-dd29-4758-9bfb-f3d233535496" path="/var/lib/kubelet/pods/0a75d8df-dd29-4758-9bfb-f3d233535496/volumes"
Sep 14 13:07:10.048295 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:10.048280    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a977a168-10fc-42f2-81dc-b91ba01edee2" path="/var/lib/kubelet/pods/a977a168-10fc-42f2-81dc-b91ba01edee2/volumes"
Sep 14 13:07:10.075390 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:10.075363    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz"]
Sep 14 13:07:10.077666 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:07:10.077646    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f0158c_fb62_4d2f_af5e_07dc2eb98205.slice/crio-9ecbe74f2ef943b93e35b3dcfa9ab3bd5e798085066127f7fe6dc80d877b8c3d WatchSource:0}: Error finding container 9ecbe74f2ef943b93e35b3dcfa9ab3bd5e798085066127f7fe6dc80d877b8c3d: Status 404 returned error can't find the container with id 9ecbe74f2ef943b93e35b3dcfa9ab3bd5e798085066127f7fe6dc80d877b8c3d
Sep 14 13:07:10.088831 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:10.088811    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-xgkvh"]
Sep 14 13:07:10.091122 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:07:10.091102    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b02f0d_9f27_4834_b312_ddc117f4433b.slice/crio-bfce6d8fa33cf8d66e6117b785fd2bcbee3e91e61f37c5fce7aba5f9d496be26 WatchSource:0}: Error finding container bfce6d8fa33cf8d66e6117b785fd2bcbee3e91e61f37c5fce7aba5f9d496be26: Status 404 returned error can't find the container with id bfce6d8fa33cf8d66e6117b785fd2bcbee3e91e61f37c5fce7aba5f9d496be26
Sep 14 13:07:10.228312 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:10.228280    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz" event={"ID":"86f0158c-fb62-4d2f-af5e-07dc2eb98205","Type":"ContainerStarted","Data":"a12662f373de2331bc41c841846229a2923b60cbc32329ac632c6f19e1e6bf4a"}
Sep 14 13:07:10.228761 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:10.228331    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz" event={"ID":"86f0158c-fb62-4d2f-af5e-07dc2eb98205","Type":"ContainerStarted","Data":"9ecbe74f2ef943b93e35b3dcfa9ab3bd5e798085066127f7fe6dc80d877b8c3d"}
Sep 14 13:07:10.228761 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:10.228401    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz"
Sep 14 13:07:10.229729 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:10.229701    2557 generic.go:356] "Generic (PLEG): container finished" podID="0a75d8df-dd29-4758-9bfb-f3d233535496" containerID="1ce39db2c6699461268afdb2f4913da2415c8e9472b0da75bf9e2f31539aec4b" exitCode=0
Sep 14 13:07:10.229867 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:10.229755    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hrqw2"
Sep 14 13:07:10.229867 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:10.229807    2557 scope.go:117] "RemoveContainer" containerID="1ce39db2c6699461268afdb2f4913da2415c8e9472b0da75bf9e2f31539aec4b"
Sep 14 13:07:10.231826 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:10.231692    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xgkvh" event={"ID":"27b02f0d-9f27-4834-b312-ddc117f4433b","Type":"ContainerStarted","Data":"2f00b8c7ef7e0279c7f1d03d4cdf01bc6b462b7cb411b50806f6ad11fd08faf0"}
Sep 14 13:07:10.231826 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:10.231783    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xgkvh" event={"ID":"27b02f0d-9f27-4834-b312-ddc117f4433b","Type":"ContainerStarted","Data":"bfce6d8fa33cf8d66e6117b785fd2bcbee3e91e61f37c5fce7aba5f9d496be26"}
Sep 14 13:07:10.231977 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:10.231921    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xgkvh"
Sep 14 13:07:10.233334 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:10.233311    2557 generic.go:356] "Generic (PLEG): container finished" podID="a977a168-10fc-42f2-81dc-b91ba01edee2" containerID="c5949b011e8f9b4fc46744d623908996e481383fa33edbf275d81711cca25cef" exitCode=0
Sep 14 13:07:10.233413 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:10.233358    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v82zf"
Sep 14 13:07:10.238506 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:10.238489    2557 scope.go:117] "RemoveContainer" containerID="1ce39db2c6699461268afdb2f4913da2415c8e9472b0da75bf9e2f31539aec4b"
Sep 14 13:07:10.238775 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:07:10.238733    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce39db2c6699461268afdb2f4913da2415c8e9472b0da75bf9e2f31539aec4b\": container with ID starting with 1ce39db2c6699461268afdb2f4913da2415c8e9472b0da75bf9e2f31539aec4b not found: ID does not exist" containerID="1ce39db2c6699461268afdb2f4913da2415c8e9472b0da75bf9e2f31539aec4b"
Sep 14 13:07:10.238858 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:10.238789    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce39db2c6699461268afdb2f4913da2415c8e9472b0da75bf9e2f31539aec4b"} err="failed to get container status \"1ce39db2c6699461268afdb2f4913da2415c8e9472b0da75bf9e2f31539aec4b\": rpc error: code = NotFound desc = could not find container \"1ce39db2c6699461268afdb2f4913da2415c8e9472b0da75bf9e2f31539aec4b\": container with ID starting with 1ce39db2c6699461268afdb2f4913da2415c8e9472b0da75bf9e2f31539aec4b not found: ID does not exist"
Sep 14 13:07:10.238858 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:10.238812    2557 scope.go:117] "RemoveContainer" containerID="c5949b011e8f9b4fc46744d623908996e481383fa33edbf275d81711cca25cef"
Sep 14 13:07:10.245959 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:10.245941    2557 scope.go:117] "RemoveContainer" containerID="c5949b011e8f9b4fc46744d623908996e481383fa33edbf275d81711cca25cef"
Sep 14 13:07:10.246199 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:07:10.246182    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5949b011e8f9b4fc46744d623908996e481383fa33edbf275d81711cca25cef\": container with ID starting with c5949b011e8f9b4fc46744d623908996e481383fa33edbf275d81711cca25cef not found: ID does not exist" containerID="c5949b011e8f9b4fc46744d623908996e481383fa33edbf275d81711cca25cef"
Sep 14 13:07:10.246261 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:10.246208    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5949b011e8f9b4fc46744d623908996e481383fa33edbf275d81711cca25cef"} err="failed to get container status \"c5949b011e8f9b4fc46744d623908996e481383fa33edbf275d81711cca25cef\": rpc error: code = NotFound desc = could not find container \"c5949b011e8f9b4fc46744d623908996e481383fa33edbf275d81711cca25cef\": container with ID starting with c5949b011e8f9b4fc46744d623908996e481383fa33edbf275d81711cca25cef not found: ID does not exist"
Sep 14 13:07:10.252306 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:10.252266    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz" podStartSLOduration=1.252253054 podStartE2EDuration="1.252253054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:07:10.249703281 +0000 UTC m=+548.684378801" watchObservedRunningTime="2026-09-14 13:07:10.252253054 +0000 UTC m=+548.686928572"
Sep 14 13:07:10.277270 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:10.277220    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xgkvh" podStartSLOduration=1.2772026140000001 podStartE2EDuration="1.277202614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:07:10.275613054 +0000 UTC m=+548.710288574" watchObservedRunningTime="2026-09-14 13:07:10.277202614 +0000 UTC m=+548.711878134"
Sep 14 13:07:11.189131 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:11.189103    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-9q2vl"
Sep 14 13:07:21.242484 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:21.242452    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz"
Sep 14 13:07:21.242917 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:21.242532    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xgkvh"
Sep 14 13:07:26.722560 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:26.722522    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz"]
Sep 14 13:07:26.723136 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:26.722729    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz" podUID="86f0158c-fb62-4d2f-af5e-07dc2eb98205" containerName="manager" containerID="cri-o://a12662f373de2331bc41c841846229a2923b60cbc32329ac632c6f19e1e6bf4a" gracePeriod=10
Sep 14 13:07:26.966206 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:26.966183    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz"
Sep 14 13:07:27.045653 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:27.045575    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86f0158c-fb62-4d2f-af5e-07dc2eb98205-extensions-socket-volume\") pod \"86f0158c-fb62-4d2f-af5e-07dc2eb98205\" (UID: \"86f0158c-fb62-4d2f-af5e-07dc2eb98205\") "
Sep 14 13:07:27.045653 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:27.045610    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brmxw\" (UniqueName: \"kubernetes.io/projected/86f0158c-fb62-4d2f-af5e-07dc2eb98205-kube-api-access-brmxw\") pod \"86f0158c-fb62-4d2f-af5e-07dc2eb98205\" (UID: \"86f0158c-fb62-4d2f-af5e-07dc2eb98205\") "
Sep 14 13:07:27.045981 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:27.045954    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f0158c-fb62-4d2f-af5e-07dc2eb98205-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "86f0158c-fb62-4d2f-af5e-07dc2eb98205" (UID: "86f0158c-fb62-4d2f-af5e-07dc2eb98205"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:07:27.047661 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:27.047631    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f0158c-fb62-4d2f-af5e-07dc2eb98205-kube-api-access-brmxw" (OuterVolumeSpecName: "kube-api-access-brmxw") pod "86f0158c-fb62-4d2f-af5e-07dc2eb98205" (UID: "86f0158c-fb62-4d2f-af5e-07dc2eb98205"). InnerVolumeSpecName "kube-api-access-brmxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:07:27.146344 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:27.146313    2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86f0158c-fb62-4d2f-af5e-07dc2eb98205-extensions-socket-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:07:27.146344 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:27.146339    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brmxw\" (UniqueName: \"kubernetes.io/projected/86f0158c-fb62-4d2f-af5e-07dc2eb98205-kube-api-access-brmxw\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:07:27.315471 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:27.315385    2557 generic.go:356] "Generic (PLEG): container finished" podID="86f0158c-fb62-4d2f-af5e-07dc2eb98205" containerID="a12662f373de2331bc41c841846229a2923b60cbc32329ac632c6f19e1e6bf4a" exitCode=0
Sep 14 13:07:27.315471 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:27.315439    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz"
Sep 14 13:07:27.315687 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:27.315443    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz" event={"ID":"86f0158c-fb62-4d2f-af5e-07dc2eb98205","Type":"ContainerDied","Data":"a12662f373de2331bc41c841846229a2923b60cbc32329ac632c6f19e1e6bf4a"}
Sep 14 13:07:27.315687 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:27.315556    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz" event={"ID":"86f0158c-fb62-4d2f-af5e-07dc2eb98205","Type":"ContainerDied","Data":"9ecbe74f2ef943b93e35b3dcfa9ab3bd5e798085066127f7fe6dc80d877b8c3d"}
Sep 14 13:07:27.315687 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:27.315578    2557 scope.go:117] "RemoveContainer" containerID="a12662f373de2331bc41c841846229a2923b60cbc32329ac632c6f19e1e6bf4a"
Sep 14 13:07:27.324547 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:27.324516    2557 scope.go:117] "RemoveContainer" containerID="a12662f373de2331bc41c841846229a2923b60cbc32329ac632c6f19e1e6bf4a"
Sep 14 13:07:27.324833 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:07:27.324815    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a12662f373de2331bc41c841846229a2923b60cbc32329ac632c6f19e1e6bf4a\": container with ID starting with a12662f373de2331bc41c841846229a2923b60cbc32329ac632c6f19e1e6bf4a not found: ID does not exist" containerID="a12662f373de2331bc41c841846229a2923b60cbc32329ac632c6f19e1e6bf4a"
Sep 14 13:07:27.324906 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:27.324842    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a12662f373de2331bc41c841846229a2923b60cbc32329ac632c6f19e1e6bf4a"} err="failed to get container status \"a12662f373de2331bc41c841846229a2923b60cbc32329ac632c6f19e1e6bf4a\": rpc error: code = NotFound desc = could not find container \"a12662f373de2331bc41c841846229a2923b60cbc32329ac632c6f19e1e6bf4a\": container with ID starting with a12662f373de2331bc41c841846229a2923b60cbc32329ac632c6f19e1e6bf4a not found: ID does not exist"
Sep 14 13:07:27.339060 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:27.339032    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz"]
Sep 14 13:07:27.342947 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:27.342925    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2dlz"]
Sep 14 13:07:28.047626 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:28.047595    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86f0158c-fb62-4d2f-af5e-07dc2eb98205" path="/var/lib/kubelet/pods/86f0158c-fb62-4d2f-af5e-07dc2eb98205/volumes"
Sep 14 13:07:43.054229 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.054197    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"]
Sep 14 13:07:43.054678 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.054631    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86f0158c-fb62-4d2f-af5e-07dc2eb98205" containerName="manager"
Sep 14 13:07:43.054678 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.054647    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="86f0158c-fb62-4d2f-af5e-07dc2eb98205" containerName="manager"
Sep 14 13:07:43.054804 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.054789    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="86f0158c-fb62-4d2f-af5e-07dc2eb98205" containerName="manager"
Sep 14 13:07:43.078395 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.078370    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"]
Sep 14 13:07:43.078546 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.078495    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.081344 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.081311    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-thtrf\""
Sep 14 13:07:43.081344 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.081311    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 14 13:07:43.158513 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.158482    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.158672 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.158521    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.158672 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.158541    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.158672 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.158557    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.158672 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.158612    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.158672 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.158643    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.158672 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.158662    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.158893 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.158681    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.158893 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.158724    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rz77\" (UniqueName: \"kubernetes.io/projected/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-kube-api-access-6rz77\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.259877 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.259842    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rz77\" (UniqueName: \"kubernetes.io/projected/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-kube-api-access-6rz77\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.260058 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.259901    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.260058 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.259946    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.260058 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.259970    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.260058 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.260011    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.260058 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.260040    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.260306 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.260078    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.260306 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.260105    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.260306 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.260190    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.260457 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.260362    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.260457 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.260381    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.260631 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.260610    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.260692 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.260636    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.260726 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.260707    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.262424 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.262403    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.262563 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.262544    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.268114 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.268083    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.268262 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.268242    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rz77\" (UniqueName: \"kubernetes.io/projected/1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60-kube-api-access-6rz77\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2ht7f\" (UID: \"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.389419 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.389347    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:43.665006 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:43.664925    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"]
Sep 14 13:07:43.668403 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:07:43.668352    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe5fabe_2ec0_4dae_9e21_4fc8639bbc60.slice/crio-45fd0eceac47f77aadb7b176d34355a839c0c0c21915186ffd0dc00fcd671815 WatchSource:0}: Error finding container 45fd0eceac47f77aadb7b176d34355a839c0c0c21915186ffd0dc00fcd671815: Status 404 returned error can't find the container with id 45fd0eceac47f77aadb7b176d34355a839c0c0c21915186ffd0dc00fcd671815
Sep 14 13:07:44.388212 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:44.388174    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f" event={"ID":"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60","Type":"ContainerStarted","Data":"45fd0eceac47f77aadb7b176d34355a839c0c0c21915186ffd0dc00fcd671815"}
Sep 14 13:07:46.275577 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:46.275541    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:07:46.275872 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:46.275615    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:07:46.275872 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:46.275644    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:07:46.401821 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:46.401784    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f" event={"ID":"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60","Type":"ContainerStarted","Data":"591bda16ab0da8f4b29a45a8cfb83cb32e5b7463d26809801cfeccf304b5b284"}
Sep 14 13:07:46.423625 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:46.423574    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f" podStartSLOduration=0.819988952 podStartE2EDuration="3.423557907s" totalImagesPullingTime="2.603568955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:07:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:07:46.421844335 +0000 UTC m=+584.856519853" watchObservedRunningTime="2026-09-14 13:07:46.423557907 +0000 UTC m=+584.858233426"
Sep 14 13:07:47.389974 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:47.389939    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:47.394451 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:47.394421    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:47.405800 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:47.405730    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:47.406678 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:47.406660    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:07:47.474013 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:47.473985    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xjzr6"]
Sep 14 13:07:47.477358 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:47.477339    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xjzr6"
Sep 14 13:07:47.480391 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:47.480372    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 14 13:07:47.487327 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:47.487304    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xjzr6"]
Sep 14 13:07:47.567308 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:47.567279    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xjzr6"]
Sep 14 13:07:47.599861 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:47.599833    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5a69db71-31de-499e-a8b6-a3785da8d5bf-config-file\") pod \"limitador-limitador-8588749977-xjzr6\" (UID: \"5a69db71-31de-499e-a8b6-a3785da8d5bf\") " pod="kuadrant-system/limitador-limitador-8588749977-xjzr6"
Sep 14 13:07:47.600006 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:47.599872    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz84m\" (UniqueName: \"kubernetes.io/projected/5a69db71-31de-499e-a8b6-a3785da8d5bf-kube-api-access-bz84m\") pod \"limitador-limitador-8588749977-xjzr6\" (UID: \"5a69db71-31de-499e-a8b6-a3785da8d5bf\") " pod="kuadrant-system/limitador-limitador-8588749977-xjzr6"
Sep 14 13:07:47.700868 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:47.700840    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5a69db71-31de-499e-a8b6-a3785da8d5bf-config-file\") pod \"limitador-limitador-8588749977-xjzr6\" (UID: \"5a69db71-31de-499e-a8b6-a3785da8d5bf\") " pod="kuadrant-system/limitador-limitador-8588749977-xjzr6"
Sep 14 13:07:47.701009 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:47.700885    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz84m\" (UniqueName: \"kubernetes.io/projected/5a69db71-31de-499e-a8b6-a3785da8d5bf-kube-api-access-bz84m\") pod \"limitador-limitador-8588749977-xjzr6\" (UID: \"5a69db71-31de-499e-a8b6-a3785da8d5bf\") " pod="kuadrant-system/limitador-limitador-8588749977-xjzr6"
Sep 14 13:07:47.701525 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:47.701507    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5a69db71-31de-499e-a8b6-a3785da8d5bf-config-file\") pod \"limitador-limitador-8588749977-xjzr6\" (UID: \"5a69db71-31de-499e-a8b6-a3785da8d5bf\") " pod="kuadrant-system/limitador-limitador-8588749977-xjzr6"
Sep 14 13:07:47.709732 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:47.709708    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz84m\" (UniqueName: \"kubernetes.io/projected/5a69db71-31de-499e-a8b6-a3785da8d5bf-kube-api-access-bz84m\") pod \"limitador-limitador-8588749977-xjzr6\" (UID: \"5a69db71-31de-499e-a8b6-a3785da8d5bf\") " pod="kuadrant-system/limitador-limitador-8588749977-xjzr6"
Sep 14 13:07:47.787700 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:47.787668    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xjzr6"
Sep 14 13:07:48.066321 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:48.064922    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xjzr6"]
Sep 14 13:07:48.279595 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:48.279554    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-6mf6p"]
Sep 14 13:07:48.284018 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:48.283998    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-6mf6p"
Sep 14 13:07:48.286605 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:48.286582    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-92tpq\""
Sep 14 13:07:48.288718 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:48.288695    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-6mf6p"]
Sep 14 13:07:48.390905 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:48.390831    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-zjf25"]
Sep 14 13:07:48.394308 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:48.394288    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-zjf25"
Sep 14 13:07:48.399015 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:48.398986    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-zjf25"]
Sep 14 13:07:48.408293 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:48.408266    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94qss\" (UniqueName: \"kubernetes.io/projected/6a97abca-4c41-4bd2-86d3-16aa50383832-kube-api-access-94qss\") pod \"authorino-7bcfd8f9cd-6mf6p\" (UID: \"6a97abca-4c41-4bd2-86d3-16aa50383832\") " pod="kuadrant-system/authorino-7bcfd8f9cd-6mf6p"
Sep 14 13:07:48.411238 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:48.411208    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xjzr6" event={"ID":"5a69db71-31de-499e-a8b6-a3785da8d5bf","Type":"ContainerStarted","Data":"466635185cd00c5a4a12a8e9f666e3dad922b199120aa4006a147b51d3523859"}
Sep 14 13:07:48.509211 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:48.509180    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw7mf\" (UniqueName: \"kubernetes.io/projected/ac78b334-5b29-4900-99ae-edf96506d0e2-kube-api-access-kw7mf\") pod \"authorino-59c974bd48-zjf25\" (UID: \"ac78b334-5b29-4900-99ae-edf96506d0e2\") " pod="kuadrant-system/authorino-59c974bd48-zjf25"
Sep 14 13:07:48.509404 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:48.509329    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94qss\" (UniqueName: \"kubernetes.io/projected/6a97abca-4c41-4bd2-86d3-16aa50383832-kube-api-access-94qss\") pod \"authorino-7bcfd8f9cd-6mf6p\" (UID: \"6a97abca-4c41-4bd2-86d3-16aa50383832\") " pod="kuadrant-system/authorino-7bcfd8f9cd-6mf6p"
Sep 14 13:07:48.517954 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:48.517923    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94qss\" (UniqueName: \"kubernetes.io/projected/6a97abca-4c41-4bd2-86d3-16aa50383832-kube-api-access-94qss\") pod \"authorino-7bcfd8f9cd-6mf6p\" (UID: \"6a97abca-4c41-4bd2-86d3-16aa50383832\") " pod="kuadrant-system/authorino-7bcfd8f9cd-6mf6p"
Sep 14 13:07:48.594683 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:48.594637    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-6mf6p"
Sep 14 13:07:48.610079 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:48.610045    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw7mf\" (UniqueName: \"kubernetes.io/projected/ac78b334-5b29-4900-99ae-edf96506d0e2-kube-api-access-kw7mf\") pod \"authorino-59c974bd48-zjf25\" (UID: \"ac78b334-5b29-4900-99ae-edf96506d0e2\") " pod="kuadrant-system/authorino-59c974bd48-zjf25"
Sep 14 13:07:48.619154 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:48.619124    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw7mf\" (UniqueName: \"kubernetes.io/projected/ac78b334-5b29-4900-99ae-edf96506d0e2-kube-api-access-kw7mf\") pod \"authorino-59c974bd48-zjf25\" (UID: \"ac78b334-5b29-4900-99ae-edf96506d0e2\") " pod="kuadrant-system/authorino-59c974bd48-zjf25"
Sep 14 13:07:48.703850 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:48.703819    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-zjf25"
Sep 14 13:07:48.862524 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:48.862483    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-6mf6p"]
Sep 14 13:07:48.865890 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:07:48.865863    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a97abca_4c41_4bd2_86d3_16aa50383832.slice/crio-e9fcfe0c68ddfb6038463127515760fc3d1fe4b97cf3ea92b155d0f3a6598cd7 WatchSource:0}: Error finding container e9fcfe0c68ddfb6038463127515760fc3d1fe4b97cf3ea92b155d0f3a6598cd7: Status 404 returned error can't find the container with id e9fcfe0c68ddfb6038463127515760fc3d1fe4b97cf3ea92b155d0f3a6598cd7
Sep 14 13:07:48.971799 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:48.971764    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-zjf25"]
Sep 14 13:07:48.974251 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:07:48.974226    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac78b334_5b29_4900_99ae_edf96506d0e2.slice/crio-b70a03f626b613005ba562f6bbf57775e0d472507c1ba705c4d646d85cd5cb1e WatchSource:0}: Error finding container b70a03f626b613005ba562f6bbf57775e0d472507c1ba705c4d646d85cd5cb1e: Status 404 returned error can't find the container with id b70a03f626b613005ba562f6bbf57775e0d472507c1ba705c4d646d85cd5cb1e
Sep 14 13:07:49.417696 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:49.417663    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-6mf6p" event={"ID":"6a97abca-4c41-4bd2-86d3-16aa50383832","Type":"ContainerStarted","Data":"e9fcfe0c68ddfb6038463127515760fc3d1fe4b97cf3ea92b155d0f3a6598cd7"}
Sep 14 13:07:49.418931 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:49.418907    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-zjf25" event={"ID":"ac78b334-5b29-4900-99ae-edf96506d0e2","Type":"ContainerStarted","Data":"b70a03f626b613005ba562f6bbf57775e0d472507c1ba705c4d646d85cd5cb1e"}
Sep 14 13:07:50.426576 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:50.426538    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xjzr6" event={"ID":"5a69db71-31de-499e-a8b6-a3785da8d5bf","Type":"ContainerStarted","Data":"e6e0aa0f5486132bfa5fd352bf5cd0383ed3d4e697ac2bc97d85ea1a53648927"}
Sep 14 13:07:50.427029 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:50.426772    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-xjzr6"
Sep 14 13:07:50.445564 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:50.445498    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-xjzr6" podStartSLOduration=2.108017632 podStartE2EDuration="3.445475616s" totalImagesPullingTime="1.337457984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:07:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:07:50.443396781 +0000 UTC m=+588.878072302" watchObservedRunningTime="2026-09-14 13:07:50.445475616 +0000 UTC m=+588.880151138"
Sep 14 13:07:51.432136 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:51.432101    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-6mf6p" event={"ID":"6a97abca-4c41-4bd2-86d3-16aa50383832","Type":"ContainerStarted","Data":"5c304bb0339176213a57972be743436539d5544ab8d21543a52c85989d01f42f"}
Sep 14 13:07:51.433627 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:51.433601    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-zjf25" event={"ID":"ac78b334-5b29-4900-99ae-edf96506d0e2","Type":"ContainerStarted","Data":"e88917dae0b98bd937cf77877960a0b2aab35fccb8b26a7bb4c6181e82d3d268"}
Sep 14 13:07:51.448263 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:51.448219    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-6mf6p" podStartSLOduration=1.485387147 podStartE2EDuration="3.448204382s" totalImagesPullingTime="1.962817235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:07:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:07:51.447022633 +0000 UTC m=+589.881698152" watchObservedRunningTime="2026-09-14 13:07:51.448204382 +0000 UTC m=+589.882879900"
Sep 14 13:07:51.465397 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:51.465345    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-zjf25" podStartSLOduration=1.615070796 podStartE2EDuration="3.465328103s" totalImagesPullingTime="1.850257307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:07:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:07:51.463684138 +0000 UTC m=+589.898359658" watchObservedRunningTime="2026-09-14 13:07:51.465328103 +0000 UTC m=+589.900003622"
Sep 14 13:07:51.485462 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:51.485433    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-6mf6p"]
Sep 14 13:07:53.442076 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:53.442037    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-6mf6p" podUID="6a97abca-4c41-4bd2-86d3-16aa50383832" containerName="authorino" containerID="cri-o://5c304bb0339176213a57972be743436539d5544ab8d21543a52c85989d01f42f" gracePeriod=30
Sep 14 13:07:53.686420 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:53.686393    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-6mf6p"
Sep 14 13:07:53.755259 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:53.755174    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94qss\" (UniqueName: \"kubernetes.io/projected/6a97abca-4c41-4bd2-86d3-16aa50383832-kube-api-access-94qss\") pod \"6a97abca-4c41-4bd2-86d3-16aa50383832\" (UID: \"6a97abca-4c41-4bd2-86d3-16aa50383832\") "
Sep 14 13:07:53.757271 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:53.757249    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a97abca-4c41-4bd2-86d3-16aa50383832-kube-api-access-94qss" (OuterVolumeSpecName: "kube-api-access-94qss") pod "6a97abca-4c41-4bd2-86d3-16aa50383832" (UID: "6a97abca-4c41-4bd2-86d3-16aa50383832"). InnerVolumeSpecName "kube-api-access-94qss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:07:53.856272 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:53.856240    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94qss\" (UniqueName: \"kubernetes.io/projected/6a97abca-4c41-4bd2-86d3-16aa50383832-kube-api-access-94qss\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:07:54.447127 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:54.447095    2557 generic.go:356] "Generic (PLEG): container finished" podID="6a97abca-4c41-4bd2-86d3-16aa50383832" containerID="5c304bb0339176213a57972be743436539d5544ab8d21543a52c85989d01f42f" exitCode=0
Sep 14 13:07:54.447603 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:54.447145    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-6mf6p"
Sep 14 13:07:54.447603 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:54.447180    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-6mf6p" event={"ID":"6a97abca-4c41-4bd2-86d3-16aa50383832","Type":"ContainerDied","Data":"5c304bb0339176213a57972be743436539d5544ab8d21543a52c85989d01f42f"}
Sep 14 13:07:54.447603 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:54.447224    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-6mf6p" event={"ID":"6a97abca-4c41-4bd2-86d3-16aa50383832","Type":"ContainerDied","Data":"e9fcfe0c68ddfb6038463127515760fc3d1fe4b97cf3ea92b155d0f3a6598cd7"}
Sep 14 13:07:54.447603 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:54.447241    2557 scope.go:117] "RemoveContainer" containerID="5c304bb0339176213a57972be743436539d5544ab8d21543a52c85989d01f42f"
Sep 14 13:07:54.455223 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:54.455189    2557 scope.go:117] "RemoveContainer" containerID="5c304bb0339176213a57972be743436539d5544ab8d21543a52c85989d01f42f"
Sep 14 13:07:54.455519 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:07:54.455499    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c304bb0339176213a57972be743436539d5544ab8d21543a52c85989d01f42f\": container with ID starting with 5c304bb0339176213a57972be743436539d5544ab8d21543a52c85989d01f42f not found: ID does not exist" containerID="5c304bb0339176213a57972be743436539d5544ab8d21543a52c85989d01f42f"
Sep 14 13:07:54.455587 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:54.455527    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c304bb0339176213a57972be743436539d5544ab8d21543a52c85989d01f42f"} err="failed to get container status \"5c304bb0339176213a57972be743436539d5544ab8d21543a52c85989d01f42f\": rpc error: code = NotFound desc = could not find container \"5c304bb0339176213a57972be743436539d5544ab8d21543a52c85989d01f42f\": container with ID starting with 5c304bb0339176213a57972be743436539d5544ab8d21543a52c85989d01f42f not found: ID does not exist"
Sep 14 13:07:54.464555 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:54.464531    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-6mf6p"]
Sep 14 13:07:54.468704 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:54.468683    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-6mf6p"]
Sep 14 13:07:56.048115 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:07:56.048085    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a97abca-4c41-4bd2-86d3-16aa50383832" path="/var/lib/kubelet/pods/6a97abca-4c41-4bd2-86d3-16aa50383832/volumes"
Sep 14 13:08:00.867482 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:00.867448    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xjzr6"]
Sep 14 13:08:00.868132 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:00.867686    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-xjzr6" podUID="5a69db71-31de-499e-a8b6-a3785da8d5bf" containerName="limitador" containerID="cri-o://e6e0aa0f5486132bfa5fd352bf5cd0383ed3d4e697ac2bc97d85ea1a53648927" gracePeriod=30
Sep 14 13:08:00.868705 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:00.868676    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-xjzr6"
Sep 14 13:08:01.413379 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:01.413359    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xjzr6"
Sep 14 13:08:01.479272 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:01.479244    2557 generic.go:356] "Generic (PLEG): container finished" podID="5a69db71-31de-499e-a8b6-a3785da8d5bf" containerID="e6e0aa0f5486132bfa5fd352bf5cd0383ed3d4e697ac2bc97d85ea1a53648927" exitCode=0
Sep 14 13:08:01.479412 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:01.479285    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xjzr6"
Sep 14 13:08:01.479412 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:01.479317    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xjzr6" event={"ID":"5a69db71-31de-499e-a8b6-a3785da8d5bf","Type":"ContainerDied","Data":"e6e0aa0f5486132bfa5fd352bf5cd0383ed3d4e697ac2bc97d85ea1a53648927"}
Sep 14 13:08:01.479412 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:01.479361    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xjzr6" event={"ID":"5a69db71-31de-499e-a8b6-a3785da8d5bf","Type":"ContainerDied","Data":"466635185cd00c5a4a12a8e9f666e3dad922b199120aa4006a147b51d3523859"}
Sep 14 13:08:01.479412 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:01.479383    2557 scope.go:117] "RemoveContainer" containerID="e6e0aa0f5486132bfa5fd352bf5cd0383ed3d4e697ac2bc97d85ea1a53648927"
Sep 14 13:08:01.487271 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:01.487253    2557 scope.go:117] "RemoveContainer" containerID="e6e0aa0f5486132bfa5fd352bf5cd0383ed3d4e697ac2bc97d85ea1a53648927"
Sep 14 13:08:01.487530 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:08:01.487514    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e0aa0f5486132bfa5fd352bf5cd0383ed3d4e697ac2bc97d85ea1a53648927\": container with ID starting with e6e0aa0f5486132bfa5fd352bf5cd0383ed3d4e697ac2bc97d85ea1a53648927 not found: ID does not exist" containerID="e6e0aa0f5486132bfa5fd352bf5cd0383ed3d4e697ac2bc97d85ea1a53648927"
Sep 14 13:08:01.487571 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:01.487538    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e0aa0f5486132bfa5fd352bf5cd0383ed3d4e697ac2bc97d85ea1a53648927"} err="failed to get container status \"e6e0aa0f5486132bfa5fd352bf5cd0383ed3d4e697ac2bc97d85ea1a53648927\": rpc error: code = NotFound desc = could not find container \"e6e0aa0f5486132bfa5fd352bf5cd0383ed3d4e697ac2bc97d85ea1a53648927\": container with ID starting with e6e0aa0f5486132bfa5fd352bf5cd0383ed3d4e697ac2bc97d85ea1a53648927 not found: ID does not exist"
Sep 14 13:08:01.517375 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:01.517350    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bz84m\" (UniqueName: \"kubernetes.io/projected/5a69db71-31de-499e-a8b6-a3785da8d5bf-kube-api-access-bz84m\") pod \"5a69db71-31de-499e-a8b6-a3785da8d5bf\" (UID: \"5a69db71-31de-499e-a8b6-a3785da8d5bf\") "
Sep 14 13:08:01.517472 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:01.517422    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5a69db71-31de-499e-a8b6-a3785da8d5bf-config-file\") pod \"5a69db71-31de-499e-a8b6-a3785da8d5bf\" (UID: \"5a69db71-31de-499e-a8b6-a3785da8d5bf\") "
Sep 14 13:08:01.517705 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:01.517685    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a69db71-31de-499e-a8b6-a3785da8d5bf-config-file" (OuterVolumeSpecName: "config-file") pod "5a69db71-31de-499e-a8b6-a3785da8d5bf" (UID: "5a69db71-31de-499e-a8b6-a3785da8d5bf"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:08:01.519336 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:01.519315    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a69db71-31de-499e-a8b6-a3785da8d5bf-kube-api-access-bz84m" (OuterVolumeSpecName: "kube-api-access-bz84m") pod "5a69db71-31de-499e-a8b6-a3785da8d5bf" (UID: "5a69db71-31de-499e-a8b6-a3785da8d5bf"). InnerVolumeSpecName "kube-api-access-bz84m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:08:01.618933 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:01.618898    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz84m\" (UniqueName: \"kubernetes.io/projected/5a69db71-31de-499e-a8b6-a3785da8d5bf-kube-api-access-bz84m\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:08:01.618933 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:01.618929    2557 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5a69db71-31de-499e-a8b6-a3785da8d5bf-config-file\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:08:01.802656 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:01.802624    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xjzr6"]
Sep 14 13:08:01.803907 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:01.803885    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xjzr6"]
Sep 14 13:08:02.017102 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:02.017069    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwqnf_62d953d6-3b86-4c95-9a47-9579d88993a7/ovn-acl-logging/0.log"
Sep 14 13:08:02.026200 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:02.026182    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwqnf_62d953d6-3b86-4c95-9a47-9579d88993a7/ovn-acl-logging/0.log"
Sep 14 13:08:02.048768 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:02.048725    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a69db71-31de-499e-a8b6-a3785da8d5bf" path="/var/lib/kubelet/pods/5a69db71-31de-499e-a8b6-a3785da8d5bf/volumes"
Sep 14 13:08:04.112342 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:04.112308    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jnsb2"]
Sep 14 13:08:04.112914 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:04.112879    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a97abca-4c41-4bd2-86d3-16aa50383832" containerName="authorino"
Sep 14 13:08:04.112914 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:04.112905    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a97abca-4c41-4bd2-86d3-16aa50383832" containerName="authorino"
Sep 14 13:08:04.113095 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:04.112932    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a69db71-31de-499e-a8b6-a3785da8d5bf" containerName="limitador"
Sep 14 13:08:04.113095 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:04.112938    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a69db71-31de-499e-a8b6-a3785da8d5bf" containerName="limitador"
Sep 14 13:08:04.113095 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:04.113003    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a97abca-4c41-4bd2-86d3-16aa50383832" containerName="authorino"
Sep 14 13:08:04.113095 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:04.113012    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a69db71-31de-499e-a8b6-a3785da8d5bf" containerName="limitador"
Sep 14 13:08:04.117550 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:04.117532    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jnsb2"
Sep 14 13:08:04.120489 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:04.120456    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-sn4ts\""
Sep 14 13:08:04.120489 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:04.120439    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 13:08:04.121805 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:04.121766    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 14 13:08:04.121805 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:04.121797    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 14 13:08:04.125861 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:04.125839    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jnsb2"]
Sep 14 13:08:04.239493 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:04.239459    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b9e50a66-5a45-46e4-abda-28c7d3e1275c-data\") pod \"postgres-69fb4594cf-jnsb2\" (UID: \"b9e50a66-5a45-46e4-abda-28c7d3e1275c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jnsb2"
Sep 14 13:08:04.239662 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:04.239508    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrhbp\" (UniqueName: \"kubernetes.io/projected/b9e50a66-5a45-46e4-abda-28c7d3e1275c-kube-api-access-vrhbp\") pod \"postgres-69fb4594cf-jnsb2\" (UID: \"b9e50a66-5a45-46e4-abda-28c7d3e1275c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jnsb2"
Sep 14 13:08:04.340696 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:04.340663    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrhbp\" (UniqueName: \"kubernetes.io/projected/b9e50a66-5a45-46e4-abda-28c7d3e1275c-kube-api-access-vrhbp\") pod \"postgres-69fb4594cf-jnsb2\" (UID: \"b9e50a66-5a45-46e4-abda-28c7d3e1275c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jnsb2"
Sep 14 13:08:04.340872 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:04.340764    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b9e50a66-5a45-46e4-abda-28c7d3e1275c-data\") pod \"postgres-69fb4594cf-jnsb2\" (UID: \"b9e50a66-5a45-46e4-abda-28c7d3e1275c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jnsb2"
Sep 14 13:08:04.341091 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:04.341076    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b9e50a66-5a45-46e4-abda-28c7d3e1275c-data\") pod \"postgres-69fb4594cf-jnsb2\" (UID: \"b9e50a66-5a45-46e4-abda-28c7d3e1275c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jnsb2"
Sep 14 13:08:04.350619 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:04.350599    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrhbp\" (UniqueName: \"kubernetes.io/projected/b9e50a66-5a45-46e4-abda-28c7d3e1275c-kube-api-access-vrhbp\") pod \"postgres-69fb4594cf-jnsb2\" (UID: \"b9e50a66-5a45-46e4-abda-28c7d3e1275c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jnsb2"
Sep 14 13:08:04.428456 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:04.428389    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jnsb2"
Sep 14 13:08:04.701639 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:04.701615    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jnsb2"]
Sep 14 13:08:04.703946 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:08:04.703919    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e50a66_5a45_46e4_abda_28c7d3e1275c.slice/crio-9fa63e0cab8406a67f6943ec624e42badf0c6a120fe24a150e66a3fbfe08e62f WatchSource:0}: Error finding container 9fa63e0cab8406a67f6943ec624e42badf0c6a120fe24a150e66a3fbfe08e62f: Status 404 returned error can't find the container with id 9fa63e0cab8406a67f6943ec624e42badf0c6a120fe24a150e66a3fbfe08e62f
Sep 14 13:08:05.501258 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:05.501221    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jnsb2" event={"ID":"b9e50a66-5a45-46e4-abda-28c7d3e1275c","Type":"ContainerStarted","Data":"9fa63e0cab8406a67f6943ec624e42badf0c6a120fe24a150e66a3fbfe08e62f"}
Sep 14 13:08:07.513468 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:07.513417    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jnsb2" event={"ID":"b9e50a66-5a45-46e4-abda-28c7d3e1275c","Type":"ContainerStarted","Data":"fb8ee07a98501feba08442e26881b8016e08eb6ecfaa996b68256e5de4e0d88b"}
Sep 14 13:08:07.513846 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:07.513513    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jnsb2"
Sep 14 13:08:07.529792 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:07.529730    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jnsb2" podStartSLOduration=1.156883197 podStartE2EDuration="3.529714098s" totalImagesPullingTime="2.372830901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:08:07.529007748 +0000 UTC m=+605.963683269" watchObservedRunningTime="2026-09-14 13:08:07.529714098 +0000 UTC m=+605.964389617"
Sep 14 13:08:13.546956 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:13.546926    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jnsb2"
Sep 14 13:08:14.095337 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.095309    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-rh9w2"]
Sep 14 13:08:14.101842 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.101821    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rh9w2"
Sep 14 13:08:14.111209 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.111184    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-rh9w2"]
Sep 14 13:08:14.221192 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.221164    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjflw\" (UniqueName: \"kubernetes.io/projected/2cbf8d71-2888-4493-a5b9-2ce993933e04-kube-api-access-zjflw\") pod \"authorino-557f876f69-rh9w2\" (UID: \"2cbf8d71-2888-4493-a5b9-2ce993933e04\") " pod="kuadrant-system/authorino-557f876f69-rh9w2"
Sep 14 13:08:14.222555 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.222535    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-rh9w2"]
Sep 14 13:08:14.222718 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:08:14.222701    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-zjflw], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-rh9w2" podUID="2cbf8d71-2888-4493-a5b9-2ce993933e04"
Sep 14 13:08:14.257103 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.257078    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-n2d2k"]
Sep 14 13:08:14.260956 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.260940    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-n2d2k"
Sep 14 13:08:14.263759 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.263709    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 14 13:08:14.268109 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.268090    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-n2d2k"]
Sep 14 13:08:14.321874 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.321856    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjflw\" (UniqueName: \"kubernetes.io/projected/2cbf8d71-2888-4493-a5b9-2ce993933e04-kube-api-access-zjflw\") pod \"authorino-557f876f69-rh9w2\" (UID: \"2cbf8d71-2888-4493-a5b9-2ce993933e04\") " pod="kuadrant-system/authorino-557f876f69-rh9w2"
Sep 14 13:08:14.332399 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.332378    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjflw\" (UniqueName: \"kubernetes.io/projected/2cbf8d71-2888-4493-a5b9-2ce993933e04-kube-api-access-zjflw\") pod \"authorino-557f876f69-rh9w2\" (UID: \"2cbf8d71-2888-4493-a5b9-2ce993933e04\") " pod="kuadrant-system/authorino-557f876f69-rh9w2"
Sep 14 13:08:14.356982 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.356926    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-n2d2k"]
Sep 14 13:08:14.357144 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:08:14.357124    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-nqccq tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-n2d2k" podUID="d7760743-9ecd-4421-b722-f70a93aa755b"
Sep 14 13:08:14.387808 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.387784    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65bb588889-l6b8h"]
Sep 14 13:08:14.392337 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.392307    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65bb588889-l6b8h"
Sep 14 13:08:14.398106 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.398086    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65bb588889-l6b8h"]
Sep 14 13:08:14.422851 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.422826    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqccq\" (UniqueName: \"kubernetes.io/projected/d7760743-9ecd-4421-b722-f70a93aa755b-kube-api-access-nqccq\") pod \"authorino-6d4dfc649f-n2d2k\" (UID: \"d7760743-9ecd-4421-b722-f70a93aa755b\") " pod="kuadrant-system/authorino-6d4dfc649f-n2d2k"
Sep 14 13:08:14.422964 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.422909    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7760743-9ecd-4421-b722-f70a93aa755b-tls-cert\") pod \"authorino-6d4dfc649f-n2d2k\" (UID: \"d7760743-9ecd-4421-b722-f70a93aa755b\") " pod="kuadrant-system/authorino-6d4dfc649f-n2d2k"
Sep 14 13:08:14.523513 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.523487    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7760743-9ecd-4421-b722-f70a93aa755b-tls-cert\") pod \"authorino-6d4dfc649f-n2d2k\" (UID: \"d7760743-9ecd-4421-b722-f70a93aa755b\") " pod="kuadrant-system/authorino-6d4dfc649f-n2d2k"
Sep 14 13:08:14.523607 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.523523    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jvzh\" (UniqueName: \"kubernetes.io/projected/ece54a39-1d5a-498d-983b-be7784c647c4-kube-api-access-5jvzh\") pod \"authorino-65bb588889-l6b8h\" (UID: \"ece54a39-1d5a-498d-983b-be7784c647c4\") " pod="kuadrant-system/authorino-65bb588889-l6b8h"
Sep 14 13:08:14.523607 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.523559    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqccq\" (UniqueName: \"kubernetes.io/projected/d7760743-9ecd-4421-b722-f70a93aa755b-kube-api-access-nqccq\") pod \"authorino-6d4dfc649f-n2d2k\" (UID: \"d7760743-9ecd-4421-b722-f70a93aa755b\") " pod="kuadrant-system/authorino-6d4dfc649f-n2d2k"
Sep 14 13:08:14.523688 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.523627    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ece54a39-1d5a-498d-983b-be7784c647c4-tls-cert\") pod \"authorino-65bb588889-l6b8h\" (UID: \"ece54a39-1d5a-498d-983b-be7784c647c4\") " pod="kuadrant-system/authorino-65bb588889-l6b8h"
Sep 14 13:08:14.525782 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.525766    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7760743-9ecd-4421-b722-f70a93aa755b-tls-cert\") pod \"authorino-6d4dfc649f-n2d2k\" (UID: \"d7760743-9ecd-4421-b722-f70a93aa755b\") " pod="kuadrant-system/authorino-6d4dfc649f-n2d2k"
Sep 14 13:08:14.534292 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.534270    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqccq\" (UniqueName: \"kubernetes.io/projected/d7760743-9ecd-4421-b722-f70a93aa755b-kube-api-access-nqccq\") pod \"authorino-6d4dfc649f-n2d2k\" (UID: \"d7760743-9ecd-4421-b722-f70a93aa755b\") " pod="kuadrant-system/authorino-6d4dfc649f-n2d2k"
Sep 14 13:08:14.543559 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.543537    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-n2d2k"
Sep 14 13:08:14.543651 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.543539    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rh9w2"
Sep 14 13:08:14.551072 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.551057    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-n2d2k"
Sep 14 13:08:14.553767 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.553752    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rh9w2"
Sep 14 13:08:14.624105 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.624058    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqccq\" (UniqueName: \"kubernetes.io/projected/d7760743-9ecd-4421-b722-f70a93aa755b-kube-api-access-nqccq\") pod \"d7760743-9ecd-4421-b722-f70a93aa755b\" (UID: \"d7760743-9ecd-4421-b722-f70a93aa755b\") "
Sep 14 13:08:14.624189 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.624117    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7760743-9ecd-4421-b722-f70a93aa755b-tls-cert\") pod \"d7760743-9ecd-4421-b722-f70a93aa755b\" (UID: \"d7760743-9ecd-4421-b722-f70a93aa755b\") "
Sep 14 13:08:14.624248 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.624195    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ece54a39-1d5a-498d-983b-be7784c647c4-tls-cert\") pod \"authorino-65bb588889-l6b8h\" (UID: \"ece54a39-1d5a-498d-983b-be7784c647c4\") " pod="kuadrant-system/authorino-65bb588889-l6b8h"
Sep 14 13:08:14.624248 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.624228    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jvzh\" (UniqueName: \"kubernetes.io/projected/ece54a39-1d5a-498d-983b-be7784c647c4-kube-api-access-5jvzh\") pod \"authorino-65bb588889-l6b8h\" (UID: \"ece54a39-1d5a-498d-983b-be7784c647c4\") " pod="kuadrant-system/authorino-65bb588889-l6b8h"
Sep 14 13:08:14.625999 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.625971    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7760743-9ecd-4421-b722-f70a93aa755b-kube-api-access-nqccq" (OuterVolumeSpecName: "kube-api-access-nqccq") pod "d7760743-9ecd-4421-b722-f70a93aa755b" (UID: "d7760743-9ecd-4421-b722-f70a93aa755b"). InnerVolumeSpecName "kube-api-access-nqccq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:08:14.626085 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.626051    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7760743-9ecd-4421-b722-f70a93aa755b-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "d7760743-9ecd-4421-b722-f70a93aa755b" (UID: "d7760743-9ecd-4421-b722-f70a93aa755b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:08:14.626416 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.626402    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ece54a39-1d5a-498d-983b-be7784c647c4-tls-cert\") pod \"authorino-65bb588889-l6b8h\" (UID: \"ece54a39-1d5a-498d-983b-be7784c647c4\") " pod="kuadrant-system/authorino-65bb588889-l6b8h"
Sep 14 13:08:14.632284 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.632267    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jvzh\" (UniqueName: \"kubernetes.io/projected/ece54a39-1d5a-498d-983b-be7784c647c4-kube-api-access-5jvzh\") pod \"authorino-65bb588889-l6b8h\" (UID: \"ece54a39-1d5a-498d-983b-be7784c647c4\") " pod="kuadrant-system/authorino-65bb588889-l6b8h"
Sep 14 13:08:14.703386 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.703366    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65bb588889-l6b8h"
Sep 14 13:08:14.725231 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.725210    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjflw\" (UniqueName: \"kubernetes.io/projected/2cbf8d71-2888-4493-a5b9-2ce993933e04-kube-api-access-zjflw\") pod \"2cbf8d71-2888-4493-a5b9-2ce993933e04\" (UID: \"2cbf8d71-2888-4493-a5b9-2ce993933e04\") "
Sep 14 13:08:14.725333 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.725321    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqccq\" (UniqueName: \"kubernetes.io/projected/d7760743-9ecd-4421-b722-f70a93aa755b-kube-api-access-nqccq\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:08:14.725396 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.725334    2557 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7760743-9ecd-4421-b722-f70a93aa755b-tls-cert\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:08:14.727038 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.727012    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cbf8d71-2888-4493-a5b9-2ce993933e04-kube-api-access-zjflw" (OuterVolumeSpecName: "kube-api-access-zjflw") pod "2cbf8d71-2888-4493-a5b9-2ce993933e04" (UID: "2cbf8d71-2888-4493-a5b9-2ce993933e04"). InnerVolumeSpecName "kube-api-access-zjflw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:08:14.825898 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.825871    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjflw\" (UniqueName: \"kubernetes.io/projected/2cbf8d71-2888-4493-a5b9-2ce993933e04-kube-api-access-zjflw\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:08:14.971919 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:14.971895    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65bb588889-l6b8h"]
Sep 14 13:08:14.974323 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:08:14.974293    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podece54a39_1d5a_498d_983b_be7784c647c4.slice/crio-13156742ffbd5f538510b85378f79d26193126d07d5e64ec51c8c928c56a1b65 WatchSource:0}: Error finding container 13156742ffbd5f538510b85378f79d26193126d07d5e64ec51c8c928c56a1b65: Status 404 returned error can't find the container with id 13156742ffbd5f538510b85378f79d26193126d07d5e64ec51c8c928c56a1b65
Sep 14 13:08:15.549547 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:15.549516    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65bb588889-l6b8h" event={"ID":"ece54a39-1d5a-498d-983b-be7784c647c4","Type":"ContainerStarted","Data":"13156742ffbd5f538510b85378f79d26193126d07d5e64ec51c8c928c56a1b65"}
Sep 14 13:08:15.549547 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:15.549545    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-n2d2k"
Sep 14 13:08:15.549726 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:15.549524    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rh9w2"
Sep 14 13:08:15.583062 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:15.583040    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-rh9w2"]
Sep 14 13:08:15.586810 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:15.586788    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-rh9w2"]
Sep 14 13:08:15.608086 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:15.608036    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-n2d2k"]
Sep 14 13:08:15.612155 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:15.612136    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-n2d2k"]
Sep 14 13:08:16.049153 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:16.049129    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cbf8d71-2888-4493-a5b9-2ce993933e04" path="/var/lib/kubelet/pods/2cbf8d71-2888-4493-a5b9-2ce993933e04/volumes"
Sep 14 13:08:16.049363 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:16.049351    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7760743-9ecd-4421-b722-f70a93aa755b" path="/var/lib/kubelet/pods/d7760743-9ecd-4421-b722-f70a93aa755b/volumes"
Sep 14 13:08:16.555481 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:16.555450    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65bb588889-l6b8h" event={"ID":"ece54a39-1d5a-498d-983b-be7784c647c4","Type":"ContainerStarted","Data":"df1dfd979f6a7a769601ec627bbe3f29bcac7e87311120b89d6485a6f9932737"}
Sep 14 13:08:16.573370 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:16.573327    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-65bb588889-l6b8h" podStartSLOduration=1.60602233 podStartE2EDuration="2.573311566s" totalImagesPullingTime="967.289236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:08:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:08:16.572767486 +0000 UTC m=+615.007442996" watchObservedRunningTime="2026-09-14 13:08:16.573311566 +0000 UTC m=+615.007987085"
Sep 14 13:08:16.601640 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:16.601610    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-zjf25"]
Sep 14 13:08:16.602063 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:16.601783    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-zjf25" podUID="ac78b334-5b29-4900-99ae-edf96506d0e2" containerName="authorino" containerID="cri-o://e88917dae0b98bd937cf77877960a0b2aab35fccb8b26a7bb4c6181e82d3d268" gracePeriod=30
Sep 14 13:08:16.843349 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:16.843322    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-zjf25"
Sep 14 13:08:16.944881 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:16.944853    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kw7mf\" (UniqueName: \"kubernetes.io/projected/ac78b334-5b29-4900-99ae-edf96506d0e2-kube-api-access-kw7mf\") pod \"ac78b334-5b29-4900-99ae-edf96506d0e2\" (UID: \"ac78b334-5b29-4900-99ae-edf96506d0e2\") "
Sep 14 13:08:16.946852 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:16.946823    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac78b334-5b29-4900-99ae-edf96506d0e2-kube-api-access-kw7mf" (OuterVolumeSpecName: "kube-api-access-kw7mf") pod "ac78b334-5b29-4900-99ae-edf96506d0e2" (UID: "ac78b334-5b29-4900-99ae-edf96506d0e2"). InnerVolumeSpecName "kube-api-access-kw7mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:08:17.045988 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:17.045953    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw7mf\" (UniqueName: \"kubernetes.io/projected/ac78b334-5b29-4900-99ae-edf96506d0e2-kube-api-access-kw7mf\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:08:17.561369 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:17.561334    2557 generic.go:356] "Generic (PLEG): container finished" podID="ac78b334-5b29-4900-99ae-edf96506d0e2" containerID="e88917dae0b98bd937cf77877960a0b2aab35fccb8b26a7bb4c6181e82d3d268" exitCode=0
Sep 14 13:08:17.561543 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:17.561375    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-zjf25"
Sep 14 13:08:17.561543 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:17.561422    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-zjf25" event={"ID":"ac78b334-5b29-4900-99ae-edf96506d0e2","Type":"ContainerDied","Data":"e88917dae0b98bd937cf77877960a0b2aab35fccb8b26a7bb4c6181e82d3d268"}
Sep 14 13:08:17.561543 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:17.561463    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-zjf25" event={"ID":"ac78b334-5b29-4900-99ae-edf96506d0e2","Type":"ContainerDied","Data":"b70a03f626b613005ba562f6bbf57775e0d472507c1ba705c4d646d85cd5cb1e"}
Sep 14 13:08:17.561543 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:17.561480    2557 scope.go:117] "RemoveContainer" containerID="e88917dae0b98bd937cf77877960a0b2aab35fccb8b26a7bb4c6181e82d3d268"
Sep 14 13:08:17.570169 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:17.570152    2557 scope.go:117] "RemoveContainer" containerID="e88917dae0b98bd937cf77877960a0b2aab35fccb8b26a7bb4c6181e82d3d268"
Sep 14 13:08:17.570433 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:08:17.570416    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e88917dae0b98bd937cf77877960a0b2aab35fccb8b26a7bb4c6181e82d3d268\": container with ID starting with e88917dae0b98bd937cf77877960a0b2aab35fccb8b26a7bb4c6181e82d3d268 not found: ID does not exist" containerID="e88917dae0b98bd937cf77877960a0b2aab35fccb8b26a7bb4c6181e82d3d268"
Sep 14 13:08:17.570482 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:17.570441    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88917dae0b98bd937cf77877960a0b2aab35fccb8b26a7bb4c6181e82d3d268"} err="failed to get container status \"e88917dae0b98bd937cf77877960a0b2aab35fccb8b26a7bb4c6181e82d3d268\": rpc error: code = NotFound desc = could not find container \"e88917dae0b98bd937cf77877960a0b2aab35fccb8b26a7bb4c6181e82d3d268\": container with ID starting with e88917dae0b98bd937cf77877960a0b2aab35fccb8b26a7bb4c6181e82d3d268 not found: ID does not exist"
Sep 14 13:08:17.587054 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:17.584610    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-zjf25"]
Sep 14 13:08:17.588937 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:17.588911    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-zjf25"]
Sep 14 13:08:18.048851 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:18.048821    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac78b334-5b29-4900-99ae-edf96506d0e2" path="/var/lib/kubelet/pods/ac78b334-5b29-4900-99ae-edf96506d0e2/volumes"
Sep 14 13:08:21.557792 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.557762    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c7c649498-zcc6d"]
Sep 14 13:08:21.558151 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.558050    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac78b334-5b29-4900-99ae-edf96506d0e2" containerName="authorino"
Sep 14 13:08:21.558151 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.558061    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac78b334-5b29-4900-99ae-edf96506d0e2" containerName="authorino"
Sep 14 13:08:21.558151 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.558124    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac78b334-5b29-4900-99ae-edf96506d0e2" containerName="authorino"
Sep 14 13:08:21.562286 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.562265    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c7c649498-zcc6d"
Sep 14 13:08:21.565099 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.565078    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 14 13:08:21.565193 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.565123    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gbc6v\""
Sep 14 13:08:21.565193 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.565176    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 14 13:08:21.565305 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.565130    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 14 13:08:21.572637 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.572615    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c7c649498-zcc6d"]
Sep 14 13:08:21.680212 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.680184    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w8qz\" (UniqueName: \"kubernetes.io/projected/a0762c04-0dac-45f9-975a-bc76b643818d-kube-api-access-4w8qz\") pod \"maas-controller-6c7c649498-zcc6d\" (UID: \"a0762c04-0dac-45f9-975a-bc76b643818d\") " pod="opendatahub/maas-controller-6c7c649498-zcc6d"
Sep 14 13:08:21.680357 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.680233    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0762c04-0dac-45f9-975a-bc76b643818d-webhook-cert\") pod \"maas-controller-6c7c649498-zcc6d\" (UID: \"a0762c04-0dac-45f9-975a-bc76b643818d\") " pod="opendatahub/maas-controller-6c7c649498-zcc6d"
Sep 14 13:08:21.680357 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.680312    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0762c04-0dac-45f9-975a-bc76b643818d-metrics-certs\") pod \"maas-controller-6c7c649498-zcc6d\" (UID: \"a0762c04-0dac-45f9-975a-bc76b643818d\") " pod="opendatahub/maas-controller-6c7c649498-zcc6d"
Sep 14 13:08:21.749553 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.749531    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c854ff6c6-rxllr"]
Sep 14 13:08:21.752654 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.752639    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c854ff6c6-rxllr"
Sep 14 13:08:21.765219 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.765198    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c854ff6c6-rxllr"]
Sep 14 13:08:21.781110 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.781087    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0762c04-0dac-45f9-975a-bc76b643818d-metrics-certs\") pod \"maas-controller-6c7c649498-zcc6d\" (UID: \"a0762c04-0dac-45f9-975a-bc76b643818d\") " pod="opendatahub/maas-controller-6c7c649498-zcc6d"
Sep 14 13:08:21.781220 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.781136    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w8qz\" (UniqueName: \"kubernetes.io/projected/a0762c04-0dac-45f9-975a-bc76b643818d-kube-api-access-4w8qz\") pod \"maas-controller-6c7c649498-zcc6d\" (UID: \"a0762c04-0dac-45f9-975a-bc76b643818d\") " pod="opendatahub/maas-controller-6c7c649498-zcc6d"
Sep 14 13:08:21.781220 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.781205    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0762c04-0dac-45f9-975a-bc76b643818d-webhook-cert\") pod \"maas-controller-6c7c649498-zcc6d\" (UID: \"a0762c04-0dac-45f9-975a-bc76b643818d\") " pod="opendatahub/maas-controller-6c7c649498-zcc6d"
Sep 14 13:08:21.783787 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.783766    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0762c04-0dac-45f9-975a-bc76b643818d-metrics-certs\") pod \"maas-controller-6c7c649498-zcc6d\" (UID: \"a0762c04-0dac-45f9-975a-bc76b643818d\") " pod="opendatahub/maas-controller-6c7c649498-zcc6d"
Sep 14 13:08:21.783897 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.783851    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0762c04-0dac-45f9-975a-bc76b643818d-webhook-cert\") pod \"maas-controller-6c7c649498-zcc6d\" (UID: \"a0762c04-0dac-45f9-975a-bc76b643818d\") " pod="opendatahub/maas-controller-6c7c649498-zcc6d"
Sep 14 13:08:21.790076 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.790054    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w8qz\" (UniqueName: \"kubernetes.io/projected/a0762c04-0dac-45f9-975a-bc76b643818d-kube-api-access-4w8qz\") pod \"maas-controller-6c7c649498-zcc6d\" (UID: \"a0762c04-0dac-45f9-975a-bc76b643818d\") " pod="opendatahub/maas-controller-6c7c649498-zcc6d"
Sep 14 13:08:21.872627 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.872565    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c7c649498-zcc6d"
Sep 14 13:08:21.881643 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.881621    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/211dcaa2-929d-44f6-8925-215ced59de6d-webhook-cert\") pod \"maas-controller-c854ff6c6-rxllr\" (UID: \"211dcaa2-929d-44f6-8925-215ced59de6d\") " pod="opendatahub/maas-controller-c854ff6c6-rxllr"
Sep 14 13:08:21.881712 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.881669    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntcn9\" (UniqueName: \"kubernetes.io/projected/211dcaa2-929d-44f6-8925-215ced59de6d-kube-api-access-ntcn9\") pod \"maas-controller-c854ff6c6-rxllr\" (UID: \"211dcaa2-929d-44f6-8925-215ced59de6d\") " pod="opendatahub/maas-controller-c854ff6c6-rxllr"
Sep 14 13:08:21.881712 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.881695    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/211dcaa2-929d-44f6-8925-215ced59de6d-metrics-certs\") pod \"maas-controller-c854ff6c6-rxllr\" (UID: \"211dcaa2-929d-44f6-8925-215ced59de6d\") " pod="opendatahub/maas-controller-c854ff6c6-rxllr"
Sep 14 13:08:21.983039 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.983010    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/211dcaa2-929d-44f6-8925-215ced59de6d-webhook-cert\") pod \"maas-controller-c854ff6c6-rxllr\" (UID: \"211dcaa2-929d-44f6-8925-215ced59de6d\") " pod="opendatahub/maas-controller-c854ff6c6-rxllr"
Sep 14 13:08:21.983209 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.983187    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntcn9\" (UniqueName: \"kubernetes.io/projected/211dcaa2-929d-44f6-8925-215ced59de6d-kube-api-access-ntcn9\") pod \"maas-controller-c854ff6c6-rxllr\" (UID: \"211dcaa2-929d-44f6-8925-215ced59de6d\") " pod="opendatahub/maas-controller-c854ff6c6-rxllr"
Sep 14 13:08:21.983284 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.983266    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/211dcaa2-929d-44f6-8925-215ced59de6d-metrics-certs\") pod \"maas-controller-c854ff6c6-rxllr\" (UID: \"211dcaa2-929d-44f6-8925-215ced59de6d\") " pod="opendatahub/maas-controller-c854ff6c6-rxllr"
Sep 14 13:08:21.985702 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.985654    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/211dcaa2-929d-44f6-8925-215ced59de6d-webhook-cert\") pod \"maas-controller-c854ff6c6-rxllr\" (UID: \"211dcaa2-929d-44f6-8925-215ced59de6d\") " pod="opendatahub/maas-controller-c854ff6c6-rxllr"
Sep 14 13:08:21.986616 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.986595    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/211dcaa2-929d-44f6-8925-215ced59de6d-metrics-certs\") pod \"maas-controller-c854ff6c6-rxllr\" (UID: \"211dcaa2-929d-44f6-8925-215ced59de6d\") " pod="opendatahub/maas-controller-c854ff6c6-rxllr"
Sep 14 13:08:21.992042 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:21.992023    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntcn9\" (UniqueName: \"kubernetes.io/projected/211dcaa2-929d-44f6-8925-215ced59de6d-kube-api-access-ntcn9\") pod \"maas-controller-c854ff6c6-rxllr\" (UID: \"211dcaa2-929d-44f6-8925-215ced59de6d\") " pod="opendatahub/maas-controller-c854ff6c6-rxllr"
Sep 14 13:08:22.062969 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:22.062944    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c854ff6c6-rxllr"
Sep 14 13:08:22.143332 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:22.143284    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c7c649498-zcc6d"]
Sep 14 13:08:22.145411 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:08:22.145385    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0762c04_0dac_45f9_975a_bc76b643818d.slice/crio-f835c664df241fb1c43b1b92ead1334ecc0f54a8b7ffa48f2d79893df1619087 WatchSource:0}: Error finding container f835c664df241fb1c43b1b92ead1334ecc0f54a8b7ffa48f2d79893df1619087: Status 404 returned error can't find the container with id f835c664df241fb1c43b1b92ead1334ecc0f54a8b7ffa48f2d79893df1619087
Sep 14 13:08:22.332501 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:22.332381    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c854ff6c6-rxllr"]
Sep 14 13:08:22.334413 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:08:22.334389    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211dcaa2_929d_44f6_8925_215ced59de6d.slice/crio-829e23cc8a957204ae7143d75e773dfee9633df6fefdd3a221b6197fc532a808 WatchSource:0}: Error finding container 829e23cc8a957204ae7143d75e773dfee9633df6fefdd3a221b6197fc532a808: Status 404 returned error can't find the container with id 829e23cc8a957204ae7143d75e773dfee9633df6fefdd3a221b6197fc532a808
Sep 14 13:08:22.585700 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:22.585662    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c854ff6c6-rxllr" event={"ID":"211dcaa2-929d-44f6-8925-215ced59de6d","Type":"ContainerStarted","Data":"829e23cc8a957204ae7143d75e773dfee9633df6fefdd3a221b6197fc532a808"}
Sep 14 13:08:22.586933 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:22.586907    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c7c649498-zcc6d" event={"ID":"a0762c04-0dac-45f9-975a-bc76b643818d","Type":"ContainerStarted","Data":"f835c664df241fb1c43b1b92ead1334ecc0f54a8b7ffa48f2d79893df1619087"}
Sep 14 13:08:25.602279 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:25.602237    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c7c649498-zcc6d" event={"ID":"a0762c04-0dac-45f9-975a-bc76b643818d","Type":"ContainerStarted","Data":"c47bd467ede6d9fb48496d68d966528703c8c9846751166586f18eb7dceb97c0"}
Sep 14 13:08:25.602754 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:25.602306    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c7c649498-zcc6d"
Sep 14 13:08:25.603618 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:25.603592    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c854ff6c6-rxllr" event={"ID":"211dcaa2-929d-44f6-8925-215ced59de6d","Type":"ContainerStarted","Data":"7e16ad01361c907c7ffa8cd59abade8597d7cb102cdaab985302c9dd1e074bf3"}
Sep 14 13:08:25.603722 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:25.603707    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c854ff6c6-rxllr"
Sep 14 13:08:25.643354 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:25.643304    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6c7c649498-zcc6d" podStartSLOduration=2.081601482 podStartE2EDuration="4.643289573s" totalImagesPullingTime="2.561688091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:08:25.642237524 +0000 UTC m=+624.076913042" watchObservedRunningTime="2026-09-14 13:08:25.643289573 +0000 UTC m=+624.077965094"
Sep 14 13:08:25.678522 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:25.678471    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c854ff6c6-rxllr" podStartSLOduration=2.306107701 podStartE2EDuration="4.678455983s" totalImagesPullingTime="2.372348282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:08:25.677246394 +0000 UTC m=+624.111921914" watchObservedRunningTime="2026-09-14 13:08:25.678455983 +0000 UTC m=+624.113131502"
Sep 14 13:08:36.611574 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:36.611547    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c854ff6c6-rxllr"
Sep 14 13:08:36.612048 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:36.611611    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c7c649498-zcc6d"
Sep 14 13:08:36.675446 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:36.675398    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c7c649498-zcc6d"]
Sep 14 13:08:36.675649 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:36.675627    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6c7c649498-zcc6d" podUID="a0762c04-0dac-45f9-975a-bc76b643818d" containerName="manager" containerID="cri-o://c47bd467ede6d9fb48496d68d966528703c8c9846751166586f18eb7dceb97c0" gracePeriod=30
Sep 14 13:08:36.909495 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:36.909473    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c7c649498-zcc6d"
Sep 14 13:08:37.000551 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:37.000525    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0762c04-0dac-45f9-975a-bc76b643818d-webhook-cert\") pod \"a0762c04-0dac-45f9-975a-bc76b643818d\" (UID: \"a0762c04-0dac-45f9-975a-bc76b643818d\") "
Sep 14 13:08:37.000679 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:37.000587    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w8qz\" (UniqueName: \"kubernetes.io/projected/a0762c04-0dac-45f9-975a-bc76b643818d-kube-api-access-4w8qz\") pod \"a0762c04-0dac-45f9-975a-bc76b643818d\" (UID: \"a0762c04-0dac-45f9-975a-bc76b643818d\") "
Sep 14 13:08:37.000679 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:37.000611    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0762c04-0dac-45f9-975a-bc76b643818d-metrics-certs\") pod \"a0762c04-0dac-45f9-975a-bc76b643818d\" (UID: \"a0762c04-0dac-45f9-975a-bc76b643818d\") "
Sep 14 13:08:37.002884 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:37.002855    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0762c04-0dac-45f9-975a-bc76b643818d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a0762c04-0dac-45f9-975a-bc76b643818d" (UID: "a0762c04-0dac-45f9-975a-bc76b643818d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:08:37.002993 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:37.002909    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0762c04-0dac-45f9-975a-bc76b643818d-kube-api-access-4w8qz" (OuterVolumeSpecName: "kube-api-access-4w8qz") pod "a0762c04-0dac-45f9-975a-bc76b643818d" (UID: "a0762c04-0dac-45f9-975a-bc76b643818d"). InnerVolumeSpecName "kube-api-access-4w8qz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:08:37.002993 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:37.002953    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0762c04-0dac-45f9-975a-bc76b643818d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a0762c04-0dac-45f9-975a-bc76b643818d" (UID: "a0762c04-0dac-45f9-975a-bc76b643818d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:08:37.101253 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:37.101226    2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0762c04-0dac-45f9-975a-bc76b643818d-metrics-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:08:37.101253 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:37.101249    2557 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0762c04-0dac-45f9-975a-bc76b643818d-webhook-cert\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:08:37.101253 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:37.101258    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w8qz\" (UniqueName: \"kubernetes.io/projected/a0762c04-0dac-45f9-975a-bc76b643818d-kube-api-access-4w8qz\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:08:37.664325 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:37.664293    2557 generic.go:356] "Generic (PLEG): container finished" podID="a0762c04-0dac-45f9-975a-bc76b643818d" containerID="c47bd467ede6d9fb48496d68d966528703c8c9846751166586f18eb7dceb97c0" exitCode=0
Sep 14 13:08:37.664803 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:37.664347    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c7c649498-zcc6d"
Sep 14 13:08:37.664803 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:37.664367    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c7c649498-zcc6d" event={"ID":"a0762c04-0dac-45f9-975a-bc76b643818d","Type":"ContainerDied","Data":"c47bd467ede6d9fb48496d68d966528703c8c9846751166586f18eb7dceb97c0"}
Sep 14 13:08:37.664803 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:37.664417    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c7c649498-zcc6d" event={"ID":"a0762c04-0dac-45f9-975a-bc76b643818d","Type":"ContainerDied","Data":"f835c664df241fb1c43b1b92ead1334ecc0f54a8b7ffa48f2d79893df1619087"}
Sep 14 13:08:37.664803 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:37.664440    2557 scope.go:117] "RemoveContainer" containerID="c47bd467ede6d9fb48496d68d966528703c8c9846751166586f18eb7dceb97c0"
Sep 14 13:08:37.674450 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:37.674432    2557 scope.go:117] "RemoveContainer" containerID="c47bd467ede6d9fb48496d68d966528703c8c9846751166586f18eb7dceb97c0"
Sep 14 13:08:37.674762 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:08:37.674719    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c47bd467ede6d9fb48496d68d966528703c8c9846751166586f18eb7dceb97c0\": container with ID starting with c47bd467ede6d9fb48496d68d966528703c8c9846751166586f18eb7dceb97c0 not found: ID does not exist" containerID="c47bd467ede6d9fb48496d68d966528703c8c9846751166586f18eb7dceb97c0"
Sep 14 13:08:37.674858 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:37.674769    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47bd467ede6d9fb48496d68d966528703c8c9846751166586f18eb7dceb97c0"} err="failed to get container status \"c47bd467ede6d9fb48496d68d966528703c8c9846751166586f18eb7dceb97c0\": rpc error: code = NotFound desc = could not find container \"c47bd467ede6d9fb48496d68d966528703c8c9846751166586f18eb7dceb97c0\": container with ID starting with c47bd467ede6d9fb48496d68d966528703c8c9846751166586f18eb7dceb97c0 not found: ID does not exist"
Sep 14 13:08:37.686169 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:37.686144    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c7c649498-zcc6d"]
Sep 14 13:08:37.692184 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:37.692162    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6c7c649498-zcc6d"]
Sep 14 13:08:38.048054 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:38.048023    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0762c04-0dac-45f9-975a-bc76b643818d" path="/var/lib/kubelet/pods/a0762c04-0dac-45f9-975a-bc76b643818d/volumes"
Sep 14 13:08:57.581424 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.581386    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-q49w6"]
Sep 14 13:08:57.581905 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.581687    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0762c04-0dac-45f9-975a-bc76b643818d" containerName="manager"
Sep 14 13:08:57.581905 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.581697    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0762c04-0dac-45f9-975a-bc76b643818d" containerName="manager"
Sep 14 13:08:57.581905 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.581768    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0762c04-0dac-45f9-975a-bc76b643818d" containerName="manager"
Sep 14 13:08:57.584627 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.584607    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q49w6"
Sep 14 13:08:57.587194 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.587172    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9zsxs\""
Sep 14 13:08:57.587292 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.587221    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 14 13:08:57.595556 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.595535    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zk2hb"]
Sep 14 13:08:57.603076 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.603045    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-q49w6"]
Sep 14 13:08:57.603179 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.603137    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-zk2hb"
Sep 14 13:08:57.611662 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.611630    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zk2hb"]
Sep 14 13:08:57.642862 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.642820    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab82241c-535a-426f-a3f3-7b902c041431-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-q49w6\" (UID: \"ab82241c-535a-426f-a3f3-7b902c041431\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q49w6"
Sep 14 13:08:57.643010 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.642883    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692bcba1-e58a-4309-a7aa-4d13e276051b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-zk2hb\" (UID: \"692bcba1-e58a-4309-a7aa-4d13e276051b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zk2hb"
Sep 14 13:08:57.643010 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.642929    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqb8d\" (UniqueName: \"kubernetes.io/projected/692bcba1-e58a-4309-a7aa-4d13e276051b-kube-api-access-rqb8d\") pod \"payload-processing-5b54b7fbb7-zk2hb\" (UID: \"692bcba1-e58a-4309-a7aa-4d13e276051b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zk2hb"
Sep 14 13:08:57.643010 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.642963    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpps8\" (UniqueName: \"kubernetes.io/projected/ab82241c-535a-426f-a3f3-7b902c041431-kube-api-access-zpps8\") pod \"payload-pre-processing-59fbc47bf8-q49w6\" (UID: \"ab82241c-535a-426f-a3f3-7b902c041431\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q49w6"
Sep 14 13:08:57.744314 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.744276    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab82241c-535a-426f-a3f3-7b902c041431-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-q49w6\" (UID: \"ab82241c-535a-426f-a3f3-7b902c041431\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q49w6"
Sep 14 13:08:57.744499 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.744327    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692bcba1-e58a-4309-a7aa-4d13e276051b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-zk2hb\" (UID: \"692bcba1-e58a-4309-a7aa-4d13e276051b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zk2hb"
Sep 14 13:08:57.744499 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.744368    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqb8d\" (UniqueName: \"kubernetes.io/projected/692bcba1-e58a-4309-a7aa-4d13e276051b-kube-api-access-rqb8d\") pod \"payload-processing-5b54b7fbb7-zk2hb\" (UID: \"692bcba1-e58a-4309-a7aa-4d13e276051b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zk2hb"
Sep 14 13:08:57.744499 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.744405    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpps8\" (UniqueName: \"kubernetes.io/projected/ab82241c-535a-426f-a3f3-7b902c041431-kube-api-access-zpps8\") pod \"payload-pre-processing-59fbc47bf8-q49w6\" (UID: \"ab82241c-535a-426f-a3f3-7b902c041431\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q49w6"
Sep 14 13:08:57.745113 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.745085    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab82241c-535a-426f-a3f3-7b902c041431-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-q49w6\" (UID: \"ab82241c-535a-426f-a3f3-7b902c041431\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q49w6"
Sep 14 13:08:57.745255 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.745231    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692bcba1-e58a-4309-a7aa-4d13e276051b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-zk2hb\" (UID: \"692bcba1-e58a-4309-a7aa-4d13e276051b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zk2hb"
Sep 14 13:08:57.755275 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.755247    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqb8d\" (UniqueName: \"kubernetes.io/projected/692bcba1-e58a-4309-a7aa-4d13e276051b-kube-api-access-rqb8d\") pod \"payload-processing-5b54b7fbb7-zk2hb\" (UID: \"692bcba1-e58a-4309-a7aa-4d13e276051b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zk2hb"
Sep 14 13:08:57.755860 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.755472    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpps8\" (UniqueName: \"kubernetes.io/projected/ab82241c-535a-426f-a3f3-7b902c041431-kube-api-access-zpps8\") pod \"payload-pre-processing-59fbc47bf8-q49w6\" (UID: \"ab82241c-535a-426f-a3f3-7b902c041431\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q49w6"
Sep 14 13:08:57.895967 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.895891    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q49w6"
Sep 14 13:08:57.914667 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:57.914627    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-zk2hb"
Sep 14 13:08:58.178256 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:58.178234    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-q49w6"]
Sep 14 13:08:58.180366 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:08:58.180333    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab82241c_535a_426f_a3f3_7b902c041431.slice/crio-5071eb94ddeec10a84daf74a46583099af2d94f753f0f9e91c2d4718c014f1ce WatchSource:0}: Error finding container 5071eb94ddeec10a84daf74a46583099af2d94f753f0f9e91c2d4718c014f1ce: Status 404 returned error can't find the container with id 5071eb94ddeec10a84daf74a46583099af2d94f753f0f9e91c2d4718c014f1ce
Sep 14 13:08:58.203771 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:58.203748    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zk2hb"]
Sep 14 13:08:58.206026 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:08:58.206004    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692bcba1_e58a_4309_a7aa_4d13e276051b.slice/crio-f4890dc76a99681d4c3f79935180d6bd4df737c592b1cdc079c1df89eafd948b WatchSource:0}: Error finding container f4890dc76a99681d4c3f79935180d6bd4df737c592b1cdc079c1df89eafd948b: Status 404 returned error can't find the container with id f4890dc76a99681d4c3f79935180d6bd4df737c592b1cdc079c1df89eafd948b
Sep 14 13:08:58.766227 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:58.766184    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q49w6" event={"ID":"ab82241c-535a-426f-a3f3-7b902c041431","Type":"ContainerStarted","Data":"5071eb94ddeec10a84daf74a46583099af2d94f753f0f9e91c2d4718c014f1ce"}
Sep 14 13:08:58.767872 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:08:58.767838    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-zk2hb" event={"ID":"692bcba1-e58a-4309-a7aa-4d13e276051b","Type":"ContainerStarted","Data":"f4890dc76a99681d4c3f79935180d6bd4df737c592b1cdc079c1df89eafd948b"}
Sep 14 13:09:01.788530 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:01.788497    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-zk2hb" event={"ID":"692bcba1-e58a-4309-a7aa-4d13e276051b","Type":"ContainerStarted","Data":"a25b36db988049a524d90f57a816218e0ae244546d9f31f2e9a8146cc0ceb433"}
Sep 14 13:09:01.789000 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:01.788572    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-zk2hb"
Sep 14 13:09:01.790424 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:01.790401    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q49w6" event={"ID":"ab82241c-535a-426f-a3f3-7b902c041431","Type":"ContainerStarted","Data":"606c32b18cdad1cc83efe0b87ad8a66cad78d2bcd0b72a6463d685bd1f4b3f82"}
Sep 14 13:09:01.790531 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:01.790514    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q49w6"
Sep 14 13:09:01.806472 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:01.806426    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-zk2hb" podStartSLOduration=1.7122983939999998 podStartE2EDuration="4.806413553s" totalImagesPullingTime="3.094115159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:08:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:09:01.805867972 +0000 UTC m=+660.240543491" watchObservedRunningTime="2026-09-14 13:09:01.806413553 +0000 UTC m=+660.241089072"
Sep 14 13:09:01.821590 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:01.821555    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q49w6" podStartSLOduration=1.695814113 podStartE2EDuration="4.821543531s" totalImagesPullingTime="3.125729418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:08:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:09:01.820481469 +0000 UTC m=+660.255156988" watchObservedRunningTime="2026-09-14 13:09:01.821543531 +0000 UTC m=+660.256219049"
Sep 14 13:09:06.817160 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:06.817134    2557 generic.go:356] "Generic (PLEG): container finished" podID="1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60" containerID="591bda16ab0da8f4b29a45a8cfb83cb32e5b7463d26809801cfeccf304b5b284" exitCode=0
Sep 14 13:09:06.817481 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:06.817208    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f" event={"ID":"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60","Type":"ContainerDied","Data":"591bda16ab0da8f4b29a45a8cfb83cb32e5b7463d26809801cfeccf304b5b284"}
Sep 14 13:09:06.817643 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:06.817626    2557 scope.go:117] "RemoveContainer" containerID="591bda16ab0da8f4b29a45a8cfb83cb32e5b7463d26809801cfeccf304b5b284"
Sep 14 13:09:06.818583 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:06.818556    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:09:06.818661 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:06.818620    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:09:06.818701 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:06.818665    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:09:07.824120 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:07.824083    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f" event={"ID":"1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60","Type":"ContainerStarted","Data":"f2469ec0531556d9acc3d118196d6635126043acaa1641678751d5bef479266e"}
Sep 14 13:09:08.390475 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:08.390438    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:09:08.390666 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:08.390633    2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2ht7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 14 13:09:08.390767 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:08.390687    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f" podUID="1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 14 13:09:09.389845 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:09.389812    2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2ht7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 14 13:09:09.390188 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:09.389880    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f" podUID="1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 14 13:09:10.390487 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:10.390455    2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2ht7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 14 13:09:10.390840 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:10.390522    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f" podUID="1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 14 13:09:11.389959 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:11.389923    2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2ht7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 14 13:09:11.390125 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:11.390004    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f" podUID="1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 14 13:09:12.390136 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:12.390095    2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2ht7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 14 13:09:12.390523 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:12.390156    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f" podUID="1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 14 13:09:12.797407 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:12.797376    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q49w6"
Sep 14 13:09:12.797775 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:12.797728    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-zk2hb"
Sep 14 13:09:13.394055 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:13.394028    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:09:13.394432 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:13.394292    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:09:13.394941 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:13.394922    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2ht7f"
Sep 14 13:09:20.790640 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:20.790605    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"]
Sep 14 13:09:20.795029 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:20.795005    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:20.799162 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:20.799139    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4m98s\""
Sep 14 13:09:20.799290 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:20.799180    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 14 13:09:20.799290 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:20.799184    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 14 13:09:20.799290 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:20.799237    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 14 13:09:20.803834 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:20.803811    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"]
Sep 14 13:09:20.973093 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:20.973058    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d80587c-6ef8-42ba-861f-b05f55b7938b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx\" (UID: \"8d80587c-6ef8-42ba-861f-b05f55b7938b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:20.973093 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:20.973092    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d80587c-6ef8-42ba-861f-b05f55b7938b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx\" (UID: \"8d80587c-6ef8-42ba-861f-b05f55b7938b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:20.973287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:20.973168    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkdc5\" (UniqueName: \"kubernetes.io/projected/8d80587c-6ef8-42ba-861f-b05f55b7938b-kube-api-access-zkdc5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx\" (UID: \"8d80587c-6ef8-42ba-861f-b05f55b7938b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:20.973287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:20.973214    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d80587c-6ef8-42ba-861f-b05f55b7938b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx\" (UID: \"8d80587c-6ef8-42ba-861f-b05f55b7938b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:20.973287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:20.973251    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d80587c-6ef8-42ba-861f-b05f55b7938b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx\" (UID: \"8d80587c-6ef8-42ba-861f-b05f55b7938b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:20.973287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:20.973269    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d80587c-6ef8-42ba-861f-b05f55b7938b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx\" (UID: \"8d80587c-6ef8-42ba-861f-b05f55b7938b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:21.074540 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:21.074456    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d80587c-6ef8-42ba-861f-b05f55b7938b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx\" (UID: \"8d80587c-6ef8-42ba-861f-b05f55b7938b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:21.074540 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:21.074496    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d80587c-6ef8-42ba-861f-b05f55b7938b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx\" (UID: \"8d80587c-6ef8-42ba-861f-b05f55b7938b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:21.074540 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:21.074529    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkdc5\" (UniqueName: \"kubernetes.io/projected/8d80587c-6ef8-42ba-861f-b05f55b7938b-kube-api-access-zkdc5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx\" (UID: \"8d80587c-6ef8-42ba-861f-b05f55b7938b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:21.074824 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:21.074558    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d80587c-6ef8-42ba-861f-b05f55b7938b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx\" (UID: \"8d80587c-6ef8-42ba-861f-b05f55b7938b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:21.074824 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:21.074598    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d80587c-6ef8-42ba-861f-b05f55b7938b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx\" (UID: \"8d80587c-6ef8-42ba-861f-b05f55b7938b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:21.074824 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:21.074626    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d80587c-6ef8-42ba-861f-b05f55b7938b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx\" (UID: \"8d80587c-6ef8-42ba-861f-b05f55b7938b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:21.074975 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:21.074958    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d80587c-6ef8-42ba-861f-b05f55b7938b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx\" (UID: \"8d80587c-6ef8-42ba-861f-b05f55b7938b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:21.075029 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:21.074988    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d80587c-6ef8-42ba-861f-b05f55b7938b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx\" (UID: \"8d80587c-6ef8-42ba-861f-b05f55b7938b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:21.075080 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:21.075043    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d80587c-6ef8-42ba-861f-b05f55b7938b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx\" (UID: \"8d80587c-6ef8-42ba-861f-b05f55b7938b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:21.076983 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:21.076962    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d80587c-6ef8-42ba-861f-b05f55b7938b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx\" (UID: \"8d80587c-6ef8-42ba-861f-b05f55b7938b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:21.077130 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:21.077108    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d80587c-6ef8-42ba-861f-b05f55b7938b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx\" (UID: \"8d80587c-6ef8-42ba-861f-b05f55b7938b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:21.082826 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:21.082802    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkdc5\" (UniqueName: \"kubernetes.io/projected/8d80587c-6ef8-42ba-861f-b05f55b7938b-kube-api-access-zkdc5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx\" (UID: \"8d80587c-6ef8-42ba-861f-b05f55b7938b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:21.106859 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:21.106827    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:21.379215 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:21.379156    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"]
Sep 14 13:09:21.381856 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:09:21.381825    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d80587c_6ef8_42ba_861f_b05f55b7938b.slice/crio-204f78eecfdc5f1947e934bdc4d604397ccfe45fd3e50fba1abdafc115d7ec84 WatchSource:0}: Error finding container 204f78eecfdc5f1947e934bdc4d604397ccfe45fd3e50fba1abdafc115d7ec84: Status 404 returned error can't find the container with id 204f78eecfdc5f1947e934bdc4d604397ccfe45fd3e50fba1abdafc115d7ec84
Sep 14 13:09:21.893994 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:21.893960    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx" event={"ID":"8d80587c-6ef8-42ba-861f-b05f55b7938b","Type":"ContainerStarted","Data":"204f78eecfdc5f1947e934bdc4d604397ccfe45fd3e50fba1abdafc115d7ec84"}
Sep 14 13:09:22.901103 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:22.901011    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx" event={"ID":"8d80587c-6ef8-42ba-861f-b05f55b7938b","Type":"ContainerStarted","Data":"5820c49f98d80e7b42e7d8261219385463e8edabe3566e8353bcc48a83005a11"}
Sep 14 13:09:22.918201 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:22.918160    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx" podStartSLOduration=1.7858228010000001 podStartE2EDuration="2.918145765s" totalImagesPullingTime="1.132322964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:09:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:09:22.917368972 +0000 UTC m=+681.352044490" watchObservedRunningTime="2026-09-14 13:09:22.918145765 +0000 UTC m=+681.352821283"
Sep 14 13:09:31.107633 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:31.107599    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:31.108213 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:31.107652    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:31.120065 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:31.120041    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:09:31.962055 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:09:31.962025    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx"
Sep 14 13:10:10.506909 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.506834    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"]
Sep 14 13:10:10.510873 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.510850    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:10.513636 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.513613    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 14 13:10:10.519878 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.519859    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"]
Sep 14 13:10:10.658625 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.658594    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85cf654b-351f-4180-a3da-a9f55f426bed-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27\" (UID: \"85cf654b-351f-4180-a3da-a9f55f426bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:10.658880 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.658638    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrdbp\" (UniqueName: \"kubernetes.io/projected/85cf654b-351f-4180-a3da-a9f55f426bed-kube-api-access-xrdbp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27\" (UID: \"85cf654b-351f-4180-a3da-a9f55f426bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:10.658880 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.658676    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85cf654b-351f-4180-a3da-a9f55f426bed-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27\" (UID: \"85cf654b-351f-4180-a3da-a9f55f426bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:10.658880 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.658754    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85cf654b-351f-4180-a3da-a9f55f426bed-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27\" (UID: \"85cf654b-351f-4180-a3da-a9f55f426bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:10.658880 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.658817    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85cf654b-351f-4180-a3da-a9f55f426bed-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27\" (UID: \"85cf654b-351f-4180-a3da-a9f55f426bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:10.658880 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.658855    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85cf654b-351f-4180-a3da-a9f55f426bed-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27\" (UID: \"85cf654b-351f-4180-a3da-a9f55f426bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:10.760010 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.759909    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85cf654b-351f-4180-a3da-a9f55f426bed-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27\" (UID: \"85cf654b-351f-4180-a3da-a9f55f426bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:10.760010 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.759961    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85cf654b-351f-4180-a3da-a9f55f426bed-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27\" (UID: \"85cf654b-351f-4180-a3da-a9f55f426bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:10.760010 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.759993    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85cf654b-351f-4180-a3da-a9f55f426bed-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27\" (UID: \"85cf654b-351f-4180-a3da-a9f55f426bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:10.760285 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.760024    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85cf654b-351f-4180-a3da-a9f55f426bed-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27\" (UID: \"85cf654b-351f-4180-a3da-a9f55f426bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:10.760285 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.760091    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85cf654b-351f-4180-a3da-a9f55f426bed-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27\" (UID: \"85cf654b-351f-4180-a3da-a9f55f426bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:10.760285 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.760114    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrdbp\" (UniqueName: \"kubernetes.io/projected/85cf654b-351f-4180-a3da-a9f55f426bed-kube-api-access-xrdbp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27\" (UID: \"85cf654b-351f-4180-a3da-a9f55f426bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:10.760439 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.760401    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85cf654b-351f-4180-a3da-a9f55f426bed-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27\" (UID: \"85cf654b-351f-4180-a3da-a9f55f426bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:10.760491 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.760448    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85cf654b-351f-4180-a3da-a9f55f426bed-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27\" (UID: \"85cf654b-351f-4180-a3da-a9f55f426bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:10.760491 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.760459    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85cf654b-351f-4180-a3da-a9f55f426bed-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27\" (UID: \"85cf654b-351f-4180-a3da-a9f55f426bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:10.762434 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.762408    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85cf654b-351f-4180-a3da-a9f55f426bed-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27\" (UID: \"85cf654b-351f-4180-a3da-a9f55f426bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:10.762579 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.762561    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85cf654b-351f-4180-a3da-a9f55f426bed-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27\" (UID: \"85cf654b-351f-4180-a3da-a9f55f426bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:10.769116 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.769088    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrdbp\" (UniqueName: \"kubernetes.io/projected/85cf654b-351f-4180-a3da-a9f55f426bed-kube-api-access-xrdbp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27\" (UID: \"85cf654b-351f-4180-a3da-a9f55f426bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:10.821768 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:10.821725    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:11.091850 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:11.091824    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"]
Sep 14 13:10:11.093915 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:10:11.093888    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85cf654b_351f_4180_a3da_a9f55f426bed.slice/crio-c5de46a607b6b977fee5c1c6e876b284b739ef61c3e5837e9dc05992b6fe94e3 WatchSource:0}: Error finding container c5de46a607b6b977fee5c1c6e876b284b739ef61c3e5837e9dc05992b6fe94e3: Status 404 returned error can't find the container with id c5de46a607b6b977fee5c1c6e876b284b739ef61c3e5837e9dc05992b6fe94e3
Sep 14 13:10:11.095207 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:11.095190    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 13:10:11.117997 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:11.117973    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27" event={"ID":"85cf654b-351f-4180-a3da-a9f55f426bed","Type":"ContainerStarted","Data":"c5de46a607b6b977fee5c1c6e876b284b739ef61c3e5837e9dc05992b6fe94e3"}
Sep 14 13:10:12.124922 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:12.124883    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27" event={"ID":"85cf654b-351f-4180-a3da-a9f55f426bed","Type":"ContainerStarted","Data":"66ae07eda2458e26ef1b49596d00eccaee3c57856e670365ed84112f7c8715a9"}
Sep 14 13:10:12.143187 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:12.143128    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27" podStartSLOduration=1.918703987 podStartE2EDuration="2.14310812s" totalImagesPullingTime="224.404133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:10:12.14297736 +0000 UTC m=+730.577652883" watchObservedRunningTime="2026-09-14 13:10:12.14310812 +0000 UTC m=+730.577783640"
Sep 14 13:10:20.822071 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:20.822028    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:20.822669 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:20.822097    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:20.834952 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:20.834927    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:10:21.179979 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:10:21.179900    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27"
Sep 14 13:11:44.421067 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:11:44.421035    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f48dbdc44-94trb"]
Sep 14 13:11:44.424525 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:11:44.424504    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f48dbdc44-94trb"
Sep 14 13:11:44.432412 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:11:44.432388    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f48dbdc44-94trb"]
Sep 14 13:11:44.564986 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:11:44.564961    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmlx4\" (UniqueName: \"kubernetes.io/projected/769edf7b-d161-4eab-bca1-8a085d0c5357-kube-api-access-kmlx4\") pod \"authorino-5f48dbdc44-94trb\" (UID: \"769edf7b-d161-4eab-bca1-8a085d0c5357\") " pod="kuadrant-system/authorino-5f48dbdc44-94trb"
Sep 14 13:11:44.565129 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:11:44.564995    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/769edf7b-d161-4eab-bca1-8a085d0c5357-tls-cert\") pod \"authorino-5f48dbdc44-94trb\" (UID: \"769edf7b-d161-4eab-bca1-8a085d0c5357\") " pod="kuadrant-system/authorino-5f48dbdc44-94trb"
Sep 14 13:11:44.665796 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:11:44.665769    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmlx4\" (UniqueName: \"kubernetes.io/projected/769edf7b-d161-4eab-bca1-8a085d0c5357-kube-api-access-kmlx4\") pod \"authorino-5f48dbdc44-94trb\" (UID: \"769edf7b-d161-4eab-bca1-8a085d0c5357\") " pod="kuadrant-system/authorino-5f48dbdc44-94trb"
Sep 14 13:11:44.665900 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:11:44.665805    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/769edf7b-d161-4eab-bca1-8a085d0c5357-tls-cert\") pod \"authorino-5f48dbdc44-94trb\" (UID: \"769edf7b-d161-4eab-bca1-8a085d0c5357\") " pod="kuadrant-system/authorino-5f48dbdc44-94trb"
Sep 14 13:11:44.668370 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:11:44.668345    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/769edf7b-d161-4eab-bca1-8a085d0c5357-tls-cert\") pod \"authorino-5f48dbdc44-94trb\" (UID: \"769edf7b-d161-4eab-bca1-8a085d0c5357\") " pod="kuadrant-system/authorino-5f48dbdc44-94trb"
Sep 14 13:11:44.676117 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:11:44.676068    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmlx4\" (UniqueName: \"kubernetes.io/projected/769edf7b-d161-4eab-bca1-8a085d0c5357-kube-api-access-kmlx4\") pod \"authorino-5f48dbdc44-94trb\" (UID: \"769edf7b-d161-4eab-bca1-8a085d0c5357\") " pod="kuadrant-system/authorino-5f48dbdc44-94trb"
Sep 14 13:11:44.734585 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:11:44.734562    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f48dbdc44-94trb"
Sep 14 13:11:45.005107 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:11:45.005079    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f48dbdc44-94trb"]
Sep 14 13:11:45.007683 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:11:45.007653    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod769edf7b_d161_4eab_bca1_8a085d0c5357.slice/crio-6b5dee155bee20e45e37056ef6a1f6ff3c7b6b779c0ab3f5dc2ef6f79e81b153 WatchSource:0}: Error finding container 6b5dee155bee20e45e37056ef6a1f6ff3c7b6b779c0ab3f5dc2ef6f79e81b153: Status 404 returned error can't find the container with id 6b5dee155bee20e45e37056ef6a1f6ff3c7b6b779c0ab3f5dc2ef6f79e81b153
Sep 14 13:11:45.555056 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:11:45.555026    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f48dbdc44-94trb" event={"ID":"769edf7b-d161-4eab-bca1-8a085d0c5357","Type":"ContainerStarted","Data":"6b5dee155bee20e45e37056ef6a1f6ff3c7b6b779c0ab3f5dc2ef6f79e81b153"}
Sep 14 13:12:00.627769 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:00.627716    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f48dbdc44-94trb" event={"ID":"769edf7b-d161-4eab-bca1-8a085d0c5357","Type":"ContainerStarted","Data":"3c22884a0559b2e5191978d6835fc176740ec1961902ef6e2beff34bc885c6bb"}
Sep 14 13:12:00.646854 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:00.646789    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5f48dbdc44-94trb" podStartSLOduration=1.807871634 podStartE2EDuration="16.646764637s" totalImagesPullingTime="14.838893003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:12:00.645055436 +0000 UTC m=+839.079730968" watchObservedRunningTime="2026-09-14 13:12:00.646764637 +0000 UTC m=+839.081440158"
Sep 14 13:12:00.678326 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:00.678291    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65bb588889-l6b8h"]
Sep 14 13:12:00.678518 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:00.678492    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-65bb588889-l6b8h" podUID="ece54a39-1d5a-498d-983b-be7784c647c4" containerName="authorino" containerID="cri-o://df1dfd979f6a7a769601ec627bbe3f29bcac7e87311120b89d6485a6f9932737" gracePeriod=30
Sep 14 13:12:00.946990 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:00.946966    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65bb588889-l6b8h"
Sep 14 13:12:01.097763 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:01.097718    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jvzh\" (UniqueName: \"kubernetes.io/projected/ece54a39-1d5a-498d-983b-be7784c647c4-kube-api-access-5jvzh\") pod \"ece54a39-1d5a-498d-983b-be7784c647c4\" (UID: \"ece54a39-1d5a-498d-983b-be7784c647c4\") "
Sep 14 13:12:01.097763 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:01.097767    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ece54a39-1d5a-498d-983b-be7784c647c4-tls-cert\") pod \"ece54a39-1d5a-498d-983b-be7784c647c4\" (UID: \"ece54a39-1d5a-498d-983b-be7784c647c4\") "
Sep 14 13:12:01.099873 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:01.099838    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece54a39-1d5a-498d-983b-be7784c647c4-kube-api-access-5jvzh" (OuterVolumeSpecName: "kube-api-access-5jvzh") pod "ece54a39-1d5a-498d-983b-be7784c647c4" (UID: "ece54a39-1d5a-498d-983b-be7784c647c4"). InnerVolumeSpecName "kube-api-access-5jvzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:12:01.107446 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:01.107420    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece54a39-1d5a-498d-983b-be7784c647c4-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ece54a39-1d5a-498d-983b-be7784c647c4" (UID: "ece54a39-1d5a-498d-983b-be7784c647c4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:12:01.198759 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:01.198588    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jvzh\" (UniqueName: \"kubernetes.io/projected/ece54a39-1d5a-498d-983b-be7784c647c4-kube-api-access-5jvzh\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:12:01.198759 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:01.198618    2557 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ece54a39-1d5a-498d-983b-be7784c647c4-tls-cert\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:12:01.634441 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:01.634356    2557 generic.go:356] "Generic (PLEG): container finished" podID="ece54a39-1d5a-498d-983b-be7784c647c4" containerID="df1dfd979f6a7a769601ec627bbe3f29bcac7e87311120b89d6485a6f9932737" exitCode=0
Sep 14 13:12:01.634441 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:01.634391    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65bb588889-l6b8h"
Sep 14 13:12:01.634441 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:01.634428    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65bb588889-l6b8h" event={"ID":"ece54a39-1d5a-498d-983b-be7784c647c4","Type":"ContainerDied","Data":"df1dfd979f6a7a769601ec627bbe3f29bcac7e87311120b89d6485a6f9932737"}
Sep 14 13:12:01.635024 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:01.634469    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65bb588889-l6b8h" event={"ID":"ece54a39-1d5a-498d-983b-be7784c647c4","Type":"ContainerDied","Data":"13156742ffbd5f538510b85378f79d26193126d07d5e64ec51c8c928c56a1b65"}
Sep 14 13:12:01.635024 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:01.634487    2557 scope.go:117] "RemoveContainer" containerID="df1dfd979f6a7a769601ec627bbe3f29bcac7e87311120b89d6485a6f9932737"
Sep 14 13:12:01.644769 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:01.644754    2557 scope.go:117] "RemoveContainer" containerID="df1dfd979f6a7a769601ec627bbe3f29bcac7e87311120b89d6485a6f9932737"
Sep 14 13:12:01.645068 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:12:01.645047    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df1dfd979f6a7a769601ec627bbe3f29bcac7e87311120b89d6485a6f9932737\": container with ID starting with df1dfd979f6a7a769601ec627bbe3f29bcac7e87311120b89d6485a6f9932737 not found: ID does not exist" containerID="df1dfd979f6a7a769601ec627bbe3f29bcac7e87311120b89d6485a6f9932737"
Sep 14 13:12:01.645136 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:01.645081    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df1dfd979f6a7a769601ec627bbe3f29bcac7e87311120b89d6485a6f9932737"} err="failed to get container status \"df1dfd979f6a7a769601ec627bbe3f29bcac7e87311120b89d6485a6f9932737\": rpc error: code = NotFound desc = could not find container \"df1dfd979f6a7a769601ec627bbe3f29bcac7e87311120b89d6485a6f9932737\": container with ID starting with df1dfd979f6a7a769601ec627bbe3f29bcac7e87311120b89d6485a6f9932737 not found: ID does not exist"
Sep 14 13:12:01.655257 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:01.655235    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65bb588889-l6b8h"]
Sep 14 13:12:01.659438 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:01.659419    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-65bb588889-l6b8h"]
Sep 14 13:12:02.049463 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:02.049433    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ece54a39-1d5a-498d-983b-be7784c647c4" path="/var/lib/kubelet/pods/ece54a39-1d5a-498d-983b-be7784c647c4/volumes"
Sep 14 13:12:17.341836 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.341802    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"]
Sep 14 13:12:17.342406 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.342230    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ece54a39-1d5a-498d-983b-be7784c647c4" containerName="authorino"
Sep 14 13:12:17.342406 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.342245    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ece54a39-1d5a-498d-983b-be7784c647c4" containerName="authorino"
Sep 14 13:12:17.342406 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.342299    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ece54a39-1d5a-498d-983b-be7784c647c4" containerName="authorino"
Sep 14 13:12:17.346776 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.346731    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.349874 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.349848    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-975227-openshift-default-dockercfg-lxsld\""
Sep 14 13:12:17.361731 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.361706    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"]
Sep 14 13:12:17.486099 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.486069    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"]
Sep 14 13:12:17.489460 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.489436    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.492796 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.492774    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b3d6c5-openshift-default-dockercfg-dsdd5\""
Sep 14 13:12:17.504836 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.504811    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"]
Sep 14 13:12:17.523338 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.523310    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b70757db-81cb-40e5-9ebb-d98486fb7f97-istiod-ca-cert\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.523492 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.523352    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-podinfo\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.523492 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.523444    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-token\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.523589 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.523491    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-data\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.523589 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.523510    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfqjv\" (UniqueName: \"kubernetes.io/projected/b70757db-81cb-40e5-9ebb-d98486fb7f97-kube-api-access-lfqjv\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.523589 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.523529    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-envoy\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.523589 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.523563    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-workload-certs\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.523789 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.523688    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-credential-socket\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.523789 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.523775    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-workload-socket\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.624360 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.624278    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6836e788-4b82-44b3-bfb9-b9718848d0c0-istiod-ca-cert\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.624360 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.624318    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-workload-certs\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.624360 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.624334    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-podinfo\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.624360 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.624355    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-token\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.624772 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.624384    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b70757db-81cb-40e5-9ebb-d98486fb7f97-istiod-ca-cert\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.624772 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.624435    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-podinfo\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.624772 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.624480    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-token\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.624772 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.624507    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-data\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.624772 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.624525    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfqjv\" (UniqueName: \"kubernetes.io/projected/b70757db-81cb-40e5-9ebb-d98486fb7f97-kube-api-access-lfqjv\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.624772 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.624545    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-envoy\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.624772 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.624573    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-envoy\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.624772 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.624606    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-workload-certs\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.624772 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.624649    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-workload-socket\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.624772 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.624680    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzpb9\" (UniqueName: \"kubernetes.io/projected/6836e788-4b82-44b3-bfb9-b9718848d0c0-kube-api-access-mzpb9\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.624772 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.624714    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-credential-socket\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.625322 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.624787    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-workload-socket\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.625322 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.624813    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-credential-socket\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.625322 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.624835    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-data\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.625322 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.625048    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-data\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.625322 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.625130    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-workload-certs\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.625322 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.625160    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b70757db-81cb-40e5-9ebb-d98486fb7f97-istiod-ca-cert\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.625322 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.625168    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-credential-socket\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.625322 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.625203    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-workload-socket\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.627048 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.627024    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-envoy\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.627147 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.627108    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-podinfo\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.645570 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.645541    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfqjv\" (UniqueName: \"kubernetes.io/projected/b70757db-81cb-40e5-9ebb-d98486fb7f97-kube-api-access-lfqjv\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.651911 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.651888    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-token\") pod \"e2e-inf-a-975227-openshift-default-77867b86c4-bls4b\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") " pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.659873 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.659857    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:17.726377 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.725930    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-credential-socket\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.726377 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.725980    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-data\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.726377 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.726026    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6836e788-4b82-44b3-bfb9-b9718848d0c0-istiod-ca-cert\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.726377 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.726056    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-workload-certs\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.726377 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.726083    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-podinfo\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.726377 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.726117    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-token\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.726377 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.726211    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-envoy\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.726377 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.726246    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-workload-socket\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.726377 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.726274    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzpb9\" (UniqueName: \"kubernetes.io/projected/6836e788-4b82-44b3-bfb9-b9718848d0c0-kube-api-access-mzpb9\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.727065 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.726407    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-credential-socket\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.727065 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.726458    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-workload-certs\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.727065 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.726485    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-data\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.727234 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.727209    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-workload-socket\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.727655 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.727612    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6836e788-4b82-44b3-bfb9-b9718848d0c0-istiod-ca-cert\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.729303 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.729275    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-envoy\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.729511 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.729492    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-podinfo\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.736130 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.736110    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-token\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.736227 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.736117    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzpb9\" (UniqueName: \"kubernetes.io/projected/6836e788-4b82-44b3-bfb9-b9718848d0c0-kube-api-access-mzpb9\") pod \"e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") " pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.804185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.804151    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:17.939143 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.939121    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"]
Sep 14 13:12:17.941658 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:12:17.941632    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70757db_81cb_40e5_9ebb_d98486fb7f97.slice/crio-6483f9b2b9e3b1d7ec412b0f12350cfcddf877ac85bd1b74a20871850863fd43 WatchSource:0}: Error finding container 6483f9b2b9e3b1d7ec412b0f12350cfcddf877ac85bd1b74a20871850863fd43: Status 404 returned error can't find the container with id 6483f9b2b9e3b1d7ec412b0f12350cfcddf877ac85bd1b74a20871850863fd43
Sep 14 13:12:17.944061 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.944022    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:12:17.944145 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.944105    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:12:17.944189 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:17.944151    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:12:18.078707 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:18.078691    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"]
Sep 14 13:12:18.081291 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:12:18.081267    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6836e788_4b82_44b3_bfb9_b9718848d0c0.slice/crio-2d4a329524dd98fbccc599ee3ada51f4a2eaf8f5bbbab58de5b3502e60f8b7c1 WatchSource:0}: Error finding container 2d4a329524dd98fbccc599ee3ada51f4a2eaf8f5bbbab58de5b3502e60f8b7c1: Status 404 returned error can't find the container with id 2d4a329524dd98fbccc599ee3ada51f4a2eaf8f5bbbab58de5b3502e60f8b7c1
Sep 14 13:12:18.083498 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:18.083471    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:12:18.083589 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:18.083525    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:12:18.083589 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:18.083549    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:12:18.717038 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:18.717000    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7" event={"ID":"6836e788-4b82-44b3-bfb9-b9718848d0c0","Type":"ContainerStarted","Data":"201ac78560540a8eb5cc24e8764b85d749e538d6453dc9566027aa627091071b"}
Sep 14 13:12:18.717416 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:18.717062    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7" event={"ID":"6836e788-4b82-44b3-bfb9-b9718848d0c0","Type":"ContainerStarted","Data":"2d4a329524dd98fbccc599ee3ada51f4a2eaf8f5bbbab58de5b3502e60f8b7c1"}
Sep 14 13:12:18.718614 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:18.718589    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b" event={"ID":"b70757db-81cb-40e5-9ebb-d98486fb7f97","Type":"ContainerStarted","Data":"c98f27bf340ae742e0ae314a9369ebc3f1d9c92f03da3b4353c5fc86c7c99a68"}
Sep 14 13:12:18.718710 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:18.718629    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b" event={"ID":"b70757db-81cb-40e5-9ebb-d98486fb7f97","Type":"ContainerStarted","Data":"6483f9b2b9e3b1d7ec412b0f12350cfcddf877ac85bd1b74a20871850863fd43"}
Sep 14 13:12:18.741105 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:18.741064    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7" podStartSLOduration=1.741051255 podStartE2EDuration="1.741051255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:12:18.738619951 +0000 UTC m=+857.173295471" watchObservedRunningTime="2026-09-14 13:12:18.741051255 +0000 UTC m=+857.175726773"
Sep 14 13:12:18.762016 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:18.761978    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b" podStartSLOduration=1.761967144 podStartE2EDuration="1.761967144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:12:18.76031896 +0000 UTC m=+857.194994480" watchObservedRunningTime="2026-09-14 13:12:18.761967144 +0000 UTC m=+857.196642659"
Sep 14 13:12:18.804301 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:18.804277    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:19.660984 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:19.660954    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:19.666254 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:19.666229    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:19.723802 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:19.723772    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:19.724394 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:19.724369    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:19.810185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:19.810153    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:20.731450 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:20.731417    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:20.732188 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:20.732172    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:22.341332 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.341302    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877"]
Sep 14 13:12:22.344948 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.344930    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877"
Sep 14 13:12:22.347947 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.347921    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-bc851d-dockercfg-7c6gh\""
Sep 14 13:12:22.348071 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.347992    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-bc851d\""
Sep 14 13:12:22.355943 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.355919    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877"]
Sep 14 13:12:22.367327 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.367303    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddm7c\" (UniqueName: \"kubernetes.io/projected/b7b603c0-b71c-4631-b269-bbd00ca584fe-kube-api-access-ddm7c\") pod \"payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877\" (UID: \"b7b603c0-b71c-4631-b269-bbd00ca584fe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877"
Sep 14 13:12:22.367454 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.367380    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7b603c0-b71c-4631-b269-bbd00ca584fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877\" (UID: \"b7b603c0-b71c-4631-b269-bbd00ca584fe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877"
Sep 14 13:12:22.373133 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.373109    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4"]
Sep 14 13:12:22.381158 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.381130    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4"
Sep 14 13:12:22.386892 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.386872    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4"]
Sep 14 13:12:22.468162 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.468130    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9lrr\" (UniqueName: \"kubernetes.io/projected/2d3c87e8-36a9-4212-89b4-7e1ba1b64b93-kube-api-access-s9lrr\") pod \"payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4\" (UID: \"2d3c87e8-36a9-4212-89b4-7e1ba1b64b93\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4"
Sep 14 13:12:22.468367 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.468187    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddm7c\" (UniqueName: \"kubernetes.io/projected/b7b603c0-b71c-4631-b269-bbd00ca584fe-kube-api-access-ddm7c\") pod \"payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877\" (UID: \"b7b603c0-b71c-4631-b269-bbd00ca584fe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877"
Sep 14 13:12:22.468367 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.468251    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3c87e8-36a9-4212-89b4-7e1ba1b64b93-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4\" (UID: \"2d3c87e8-36a9-4212-89b4-7e1ba1b64b93\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4"
Sep 14 13:12:22.468367 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.468309    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7b603c0-b71c-4631-b269-bbd00ca584fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877\" (UID: \"b7b603c0-b71c-4631-b269-bbd00ca584fe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877"
Sep 14 13:12:22.469082 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.469061    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7b603c0-b71c-4631-b269-bbd00ca584fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877\" (UID: \"b7b603c0-b71c-4631-b269-bbd00ca584fe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877"
Sep 14 13:12:22.481487 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.481465    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddm7c\" (UniqueName: \"kubernetes.io/projected/b7b603c0-b71c-4631-b269-bbd00ca584fe-kube-api-access-ddm7c\") pod \"payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877\" (UID: \"b7b603c0-b71c-4631-b269-bbd00ca584fe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877"
Sep 14 13:12:22.569080 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.569039    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9lrr\" (UniqueName: \"kubernetes.io/projected/2d3c87e8-36a9-4212-89b4-7e1ba1b64b93-kube-api-access-s9lrr\") pod \"payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4\" (UID: \"2d3c87e8-36a9-4212-89b4-7e1ba1b64b93\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4"
Sep 14 13:12:22.569367 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.569103    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3c87e8-36a9-4212-89b4-7e1ba1b64b93-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4\" (UID: \"2d3c87e8-36a9-4212-89b4-7e1ba1b64b93\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4"
Sep 14 13:12:22.569891 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.569867    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3c87e8-36a9-4212-89b4-7e1ba1b64b93-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4\" (UID: \"2d3c87e8-36a9-4212-89b4-7e1ba1b64b93\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4"
Sep 14 13:12:22.577908 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.577886    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9lrr\" (UniqueName: \"kubernetes.io/projected/2d3c87e8-36a9-4212-89b4-7e1ba1b64b93-kube-api-access-s9lrr\") pod \"payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4\" (UID: \"2d3c87e8-36a9-4212-89b4-7e1ba1b64b93\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4"
Sep 14 13:12:22.658045 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.657973    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877"
Sep 14 13:12:22.692478 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.692448    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4"
Sep 14 13:12:22.930697 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.930672    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877"]
Sep 14 13:12:22.932125 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:12:22.932099    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b603c0_b71c_4631_b269_bbd00ca584fe.slice/crio-7f6329388207702251cc120b3a2be0c1913afc5e16a928bc2962eaad51036cca WatchSource:0}: Error finding container 7f6329388207702251cc120b3a2be0c1913afc5e16a928bc2962eaad51036cca: Status 404 returned error can't find the container with id 7f6329388207702251cc120b3a2be0c1913afc5e16a928bc2962eaad51036cca
Sep 14 13:12:22.987698 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:22.987670    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4"]
Sep 14 13:12:22.990143 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:12:22.990120    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3c87e8_36a9_4212_89b4_7e1ba1b64b93.slice/crio-a379b9aa44e9dc312ff3206e79dd695e862bc34e8a1e5ee985b70795735cebbe WatchSource:0}: Error finding container a379b9aa44e9dc312ff3206e79dd695e862bc34e8a1e5ee985b70795735cebbe: Status 404 returned error can't find the container with id a379b9aa44e9dc312ff3206e79dd695e862bc34e8a1e5ee985b70795735cebbe
Sep 14 13:12:23.367247 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.367175    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr"]
Sep 14 13:12:23.371332 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.371310    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr"
Sep 14 13:12:23.374147 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.374129    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-b3d6c5-dockercfg-lprf8\""
Sep 14 13:12:23.374351 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.374330    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-b3d6c5\""
Sep 14 13:12:23.374562 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.374544    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad7cd32-9384-45f1-bcbd-bcdb5ae40361-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr\" (UID: \"fad7cd32-9384-45f1-bcbd-bcdb5ae40361\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr"
Sep 14 13:12:23.374669 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.374580    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzrp7\" (UniqueName: \"kubernetes.io/projected/fad7cd32-9384-45f1-bcbd-bcdb5ae40361-kube-api-access-kzrp7\") pod \"payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr\" (UID: \"fad7cd32-9384-45f1-bcbd-bcdb5ae40361\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr"
Sep 14 13:12:23.384158 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.384139    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr"]
Sep 14 13:12:23.391849 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.391829    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n"]
Sep 14 13:12:23.395140 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.395124    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n"
Sep 14 13:12:23.413242 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.413223    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n"]
Sep 14 13:12:23.475328 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.475299    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/876b1975-7474-4482-8551-da99ab4a1f26-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n\" (UID: \"876b1975-7474-4482-8551-da99ab4a1f26\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n"
Sep 14 13:12:23.475641 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.475622    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad7cd32-9384-45f1-bcbd-bcdb5ae40361-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr\" (UID: \"fad7cd32-9384-45f1-bcbd-bcdb5ae40361\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr"
Sep 14 13:12:23.476700 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.476679    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzrp7\" (UniqueName: \"kubernetes.io/projected/fad7cd32-9384-45f1-bcbd-bcdb5ae40361-kube-api-access-kzrp7\") pod \"payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr\" (UID: \"fad7cd32-9384-45f1-bcbd-bcdb5ae40361\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr"
Sep 14 13:12:23.476940 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.476923    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-529nz\" (UniqueName: \"kubernetes.io/projected/876b1975-7474-4482-8551-da99ab4a1f26-kube-api-access-529nz\") pod \"payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n\" (UID: \"876b1975-7474-4482-8551-da99ab4a1f26\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n"
Sep 14 13:12:23.477137 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.476615    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad7cd32-9384-45f1-bcbd-bcdb5ae40361-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr\" (UID: \"fad7cd32-9384-45f1-bcbd-bcdb5ae40361\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr"
Sep 14 13:12:23.492378 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.492206    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzrp7\" (UniqueName: \"kubernetes.io/projected/fad7cd32-9384-45f1-bcbd-bcdb5ae40361-kube-api-access-kzrp7\") pod \"payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr\" (UID: \"fad7cd32-9384-45f1-bcbd-bcdb5ae40361\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr"
Sep 14 13:12:23.577525 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.577486    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-529nz\" (UniqueName: \"kubernetes.io/projected/876b1975-7474-4482-8551-da99ab4a1f26-kube-api-access-529nz\") pod \"payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n\" (UID: \"876b1975-7474-4482-8551-da99ab4a1f26\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n"
Sep 14 13:12:23.577709 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.577573    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/876b1975-7474-4482-8551-da99ab4a1f26-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n\" (UID: \"876b1975-7474-4482-8551-da99ab4a1f26\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n"
Sep 14 13:12:23.578364 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.578340    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/876b1975-7474-4482-8551-da99ab4a1f26-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n\" (UID: \"876b1975-7474-4482-8551-da99ab4a1f26\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n"
Sep 14 13:12:23.587837 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.587810    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-529nz\" (UniqueName: \"kubernetes.io/projected/876b1975-7474-4482-8551-da99ab4a1f26-kube-api-access-529nz\") pod \"payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n\" (UID: \"876b1975-7474-4482-8551-da99ab4a1f26\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n"
Sep 14 13:12:23.683454 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.683372    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr"
Sep 14 13:12:23.705208 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.705182    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n"
Sep 14 13:12:23.758149 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.756321    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877" event={"ID":"b7b603c0-b71c-4631-b269-bbd00ca584fe","Type":"ContainerStarted","Data":"b61febad8587bc45b9251c477e2fa7634ce24eefffdabc2a499dbaeb2d5d3e75"}
Sep 14 13:12:23.758149 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.756384    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877" event={"ID":"b7b603c0-b71c-4631-b269-bbd00ca584fe","Type":"ContainerStarted","Data":"7f6329388207702251cc120b3a2be0c1913afc5e16a928bc2962eaad51036cca"}
Sep 14 13:12:23.758149 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.758088    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877"
Sep 14 13:12:23.760125 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.759493    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4" event={"ID":"2d3c87e8-36a9-4212-89b4-7e1ba1b64b93","Type":"ContainerStarted","Data":"a2843489ca9fdb173b509462c191f34b6d176397761ce07563d77f23d389c71f"}
Sep 14 13:12:23.760125 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.759540    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4" event={"ID":"2d3c87e8-36a9-4212-89b4-7e1ba1b64b93","Type":"ContainerStarted","Data":"a379b9aa44e9dc312ff3206e79dd695e862bc34e8a1e5ee985b70795735cebbe"}
Sep 14 13:12:23.760125 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.760047    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4"
Sep 14 13:12:23.795046 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.794609    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877" podStartSLOduration=1.7945847160000001 podStartE2EDuration="1.794584716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:12:23.792488478 +0000 UTC m=+862.227163990" watchObservedRunningTime="2026-09-14 13:12:23.794584716 +0000 UTC m=+862.229260236"
Sep 14 13:12:23.834372 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.834308    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4" podStartSLOduration=1.834289444 podStartE2EDuration="1.834289444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:12:23.832753913 +0000 UTC m=+862.267429423" watchObservedRunningTime="2026-09-14 13:12:23.834289444 +0000 UTC m=+862.268964963"
Sep 14 13:12:23.968133 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.968106    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr"]
Sep 14 13:12:23.969614 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:12:23.969583    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad7cd32_9384_45f1_bcbd_bcdb5ae40361.slice/crio-e2f96e6e37621f377e6bd6ccbbaa121580ed135e9e7231e65d9d36eb0c483689 WatchSource:0}: Error finding container e2f96e6e37621f377e6bd6ccbbaa121580ed135e9e7231e65d9d36eb0c483689: Status 404 returned error can't find the container with id e2f96e6e37621f377e6bd6ccbbaa121580ed135e9e7231e65d9d36eb0c483689
Sep 14 13:12:23.998651 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:23.998630    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n"]
Sep 14 13:12:24.000905 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:12:24.000879    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod876b1975_7474_4482_8551_da99ab4a1f26.slice/crio-eef0090433f9a99d70e5fdfd489c8933db43f71024a008212ea409ba76dcd105 WatchSource:0}: Error finding container eef0090433f9a99d70e5fdfd489c8933db43f71024a008212ea409ba76dcd105: Status 404 returned error can't find the container with id eef0090433f9a99d70e5fdfd489c8933db43f71024a008212ea409ba76dcd105
Sep 14 13:12:24.768839 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:24.768796    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n" event={"ID":"876b1975-7474-4482-8551-da99ab4a1f26","Type":"ContainerStarted","Data":"433dc05de1b425b9d224d69c66d3506e43abb54b37059124ec940f5a45bdd6e2"}
Sep 14 13:12:24.769308 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:24.768915    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n" event={"ID":"876b1975-7474-4482-8551-da99ab4a1f26","Type":"ContainerStarted","Data":"eef0090433f9a99d70e5fdfd489c8933db43f71024a008212ea409ba76dcd105"}
Sep 14 13:12:24.769308 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:24.768969    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n"
Sep 14 13:12:24.771049 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:24.771018    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr" event={"ID":"fad7cd32-9384-45f1-bcbd-bcdb5ae40361","Type":"ContainerStarted","Data":"a985139d425b07f0d5442a735cca84d76b9890db9252b491a2e5152ddbda976a"}
Sep 14 13:12:24.771186 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:24.771074    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr" event={"ID":"fad7cd32-9384-45f1-bcbd-bcdb5ae40361","Type":"ContainerStarted","Data":"e2f96e6e37621f377e6bd6ccbbaa121580ed135e9e7231e65d9d36eb0c483689"}
Sep 14 13:12:24.771377 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:24.771357    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr"
Sep 14 13:12:24.787681 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:24.787637    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n" podStartSLOduration=1.7876221650000002 podStartE2EDuration="1.787622165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:12:24.785973744 +0000 UTC m=+863.220649274" watchObservedRunningTime="2026-09-14 13:12:24.787622165 +0000 UTC m=+863.222297685"
Sep 14 13:12:24.803385 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:24.803333    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr" podStartSLOduration=1.803313216 podStartE2EDuration="1.803313216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:12:24.802645849 +0000 UTC m=+863.237321369" watchObservedRunningTime="2026-09-14 13:12:24.803313216 +0000 UTC m=+863.237988737"
Sep 14 13:12:25.445653 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.445621    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"]
Sep 14 13:12:25.449688 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.449664    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.452750 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.452715    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-c14a5e-openshift-default-dockercfg-jnsgl\""
Sep 14 13:12:25.462630 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.462606    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"]
Sep 14 13:12:25.497121 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.497087    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-podinfo\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.497121 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.497117    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8110160b-b752-4f51-ab9a-a76e74aa919c-istiod-ca-cert\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.497353 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.497175    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-workload-socket\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.497353 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.497217    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-workload-certs\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.497353 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.497243    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-data\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.497353 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.497262    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-token\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.497353 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.497343    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-credential-socket\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.497595 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.497401    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k769s\" (UniqueName: \"kubernetes.io/projected/8110160b-b752-4f51-ab9a-a76e74aa919c-kube-api-access-k769s\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.497595 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.497421    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-envoy\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.598529 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.598489    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-workload-socket\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.598715 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.598541    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-workload-certs\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.598715 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.598575    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-data\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.598715 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.598598    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-token\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.598715 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.598649    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-credential-socket\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.598914 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.598860    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k769s\" (UniqueName: \"kubernetes.io/projected/8110160b-b752-4f51-ab9a-a76e74aa919c-kube-api-access-k769s\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.598965 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.598912    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-envoy\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.599016 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.598962    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-workload-socket\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.599016 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.598987    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-workload-certs\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.599016 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.598965    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-podinfo\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.599159 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.599066    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8110160b-b752-4f51-ab9a-a76e74aa919c-istiod-ca-cert\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.599159 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.599122    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-data\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.599251 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.599230    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-credential-socket\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.599642 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.599621    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8110160b-b752-4f51-ab9a-a76e74aa919c-istiod-ca-cert\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.601081 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.601055    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-envoy\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.601363 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.601345    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-podinfo\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.606642 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.606620    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-token\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.606815 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.606799    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k769s\" (UniqueName: \"kubernetes.io/projected/8110160b-b752-4f51-ab9a-a76e74aa919c-kube-api-access-k769s\") pod \"e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") " pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:25.762392 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:25.762306    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:26.040424 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:26.040336    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"]
Sep 14 13:12:26.043792 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:12:26.043758    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8110160b_b752_4f51_ab9a_a76e74aa919c.slice/crio-7a0ec9354ea0514dae8257c57aa058e0466aba27eadaea24620f363a60ad280e WatchSource:0}: Error finding container 7a0ec9354ea0514dae8257c57aa058e0466aba27eadaea24620f363a60ad280e: Status 404 returned error can't find the container with id 7a0ec9354ea0514dae8257c57aa058e0466aba27eadaea24620f363a60ad280e
Sep 14 13:12:26.046187 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:26.046158    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:12:26.046256 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:26.046220    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:12:26.046256 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:26.046246    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:12:26.784510 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:26.784472    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq" event={"ID":"8110160b-b752-4f51-ab9a-a76e74aa919c","Type":"ContainerStarted","Data":"23b2e7479beed2f36dac93743ea0121538d09128d38d791e4ed5495f5e2942ff"}
Sep 14 13:12:26.784655 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:26.784533    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq" event={"ID":"8110160b-b752-4f51-ab9a-a76e74aa919c","Type":"ContainerStarted","Data":"7a0ec9354ea0514dae8257c57aa058e0466aba27eadaea24620f363a60ad280e"}
Sep 14 13:12:26.806890 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:26.806846    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq" podStartSLOduration=1.806830659 podStartE2EDuration="1.806830659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:12:26.80482012 +0000 UTC m=+865.239495640" watchObservedRunningTime="2026-09-14 13:12:26.806830659 +0000 UTC m=+865.241506174"
Sep 14 13:12:27.763283 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:27.763252    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:27.768476 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:27.768446    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:27.790371 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:27.790336    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:27.791195 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:27.791174    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:33.634618 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:33.634590    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms"]
Sep 14 13:12:33.638369 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:33.638348    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms"
Sep 14 13:12:33.642071 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:33.642041    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-360610b9-dockercfg-2rps9\""
Sep 14 13:12:33.642297 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:33.642276    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-360610b9\""
Sep 14 13:12:33.657761 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:33.653595    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms"]
Sep 14 13:12:33.696762 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:33.696359    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c02f172-6c1b-4215-8602-2ad5740c0b6c-plugins-config-volume\") pod \"payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms\" (UID: \"6c02f172-6c1b-4215-8602-2ad5740c0b6c\") " pod="openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms"
Sep 14 13:12:33.696762 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:33.696429    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spd27\" (UniqueName: \"kubernetes.io/projected/6c02f172-6c1b-4215-8602-2ad5740c0b6c-kube-api-access-spd27\") pod \"payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms\" (UID: \"6c02f172-6c1b-4215-8602-2ad5740c0b6c\") " pod="openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms"
Sep 14 13:12:33.799312 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:33.799261    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c02f172-6c1b-4215-8602-2ad5740c0b6c-plugins-config-volume\") pod \"payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms\" (UID: \"6c02f172-6c1b-4215-8602-2ad5740c0b6c\") " pod="openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms"
Sep 14 13:12:33.800490 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:33.800468    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spd27\" (UniqueName: \"kubernetes.io/projected/6c02f172-6c1b-4215-8602-2ad5740c0b6c-kube-api-access-spd27\") pod \"payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms\" (UID: \"6c02f172-6c1b-4215-8602-2ad5740c0b6c\") " pod="openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms"
Sep 14 13:12:33.801093 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:33.800406    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c02f172-6c1b-4215-8602-2ad5740c0b6c-plugins-config-volume\") pod \"payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms\" (UID: \"6c02f172-6c1b-4215-8602-2ad5740c0b6c\") " pod="openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms"
Sep 14 13:12:33.810567 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:33.810541    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spd27\" (UniqueName: \"kubernetes.io/projected/6c02f172-6c1b-4215-8602-2ad5740c0b6c-kube-api-access-spd27\") pod \"payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms\" (UID: \"6c02f172-6c1b-4215-8602-2ad5740c0b6c\") " pod="openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms"
Sep 14 13:12:33.954760 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:33.954708    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms"
Sep 14 13:12:34.346192 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:34.343226    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms"]
Sep 14 13:12:34.827537 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:34.827503    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms" event={"ID":"6c02f172-6c1b-4215-8602-2ad5740c0b6c","Type":"ContainerStarted","Data":"a7bd6d0144b910a125132bd2f005de0b087330ae41a530628bdef598a2eb43c4"}
Sep 14 13:12:34.827935 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:34.827556    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms" event={"ID":"6c02f172-6c1b-4215-8602-2ad5740c0b6c","Type":"ContainerStarted","Data":"f27c59115e2de3fbb4df9ecaa7fd46277eb184de364e6be335a33c44d1f9f2b0"}
Sep 14 13:12:34.827935 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:34.827687    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms"
Sep 14 13:12:35.023796 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:35.023728    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms" podStartSLOduration=2.023709055 podStartE2EDuration="2.023709055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:12:34.865007535 +0000 UTC m=+873.299683066" watchObservedRunningTime="2026-09-14 13:12:35.023709055 +0000 UTC m=+873.458384573"
Sep 14 13:12:35.025492 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:35.025473    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv"]
Sep 14 13:12:35.029183 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:35.029168    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv"
Sep 14 13:12:35.032016 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:35.031990    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-c14a5e-dockercfg-8mt9k\""
Sep 14 13:12:35.032123 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:35.032021    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-c14a5e\""
Sep 14 13:12:35.039276 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:35.039256    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv"]
Sep 14 13:12:35.121928 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:35.121837    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6-plugins-config-volume\") pod \"payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv\" (UID: \"2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6\") " pod="openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv"
Sep 14 13:12:35.121928 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:35.121897    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmnzn\" (UniqueName: \"kubernetes.io/projected/2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6-kube-api-access-rmnzn\") pod \"payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv\" (UID: \"2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6\") " pod="openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv"
Sep 14 13:12:35.222907 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:35.222865    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6-plugins-config-volume\") pod \"payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv\" (UID: \"2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6\") " pod="openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv"
Sep 14 13:12:35.223098 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:35.222971    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmnzn\" (UniqueName: \"kubernetes.io/projected/2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6-kube-api-access-rmnzn\") pod \"payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv\" (UID: \"2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6\") " pod="openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv"
Sep 14 13:12:35.223835 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:35.223812    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6-plugins-config-volume\") pod \"payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv\" (UID: \"2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6\") " pod="openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv"
Sep 14 13:12:35.242946 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:35.242915    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmnzn\" (UniqueName: \"kubernetes.io/projected/2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6-kube-api-access-rmnzn\") pod \"payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv\" (UID: \"2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6\") " pod="openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv"
Sep 14 13:12:35.340275 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:35.340235    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv"
Sep 14 13:12:35.656811 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:35.653525    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv"]
Sep 14 13:12:35.664870 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:12:35.664836    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9b0ef4_0f67_4a47_ad16_fc09fe0748d6.slice/crio-2d705d6b25971004b7ebfce05a3115b9a01bb856011117fab58c73faac7487ce WatchSource:0}: Error finding container 2d705d6b25971004b7ebfce05a3115b9a01bb856011117fab58c73faac7487ce: Status 404 returned error can't find the container with id 2d705d6b25971004b7ebfce05a3115b9a01bb856011117fab58c73faac7487ce
Sep 14 13:12:35.783282 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:35.783154    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n"
Sep 14 13:12:35.783282 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:35.783246    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4"
Sep 14 13:12:35.783509 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:35.783298    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr"
Sep 14 13:12:35.784237 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:35.784088    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877"
Sep 14 13:12:35.837719 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:35.837619    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv" event={"ID":"2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6","Type":"ContainerStarted","Data":"f9f59cc2f09aafe2278ff40054ba053aaf5197537e5ef84a29091ee4af675f2c"}
Sep 14 13:12:35.837719 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:35.837684    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv" event={"ID":"2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6","Type":"ContainerStarted","Data":"2d705d6b25971004b7ebfce05a3115b9a01bb856011117fab58c73faac7487ce"}
Sep 14 13:12:35.838345 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:35.838289    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv"
Sep 14 13:12:35.918642 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:35.918504    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv" podStartSLOduration=0.918480259 podStartE2EDuration="918.480259ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:12:35.915669075 +0000 UTC m=+874.350344595" watchObservedRunningTime="2026-09-14 13:12:35.918480259 +0000 UTC m=+874.353155783"
Sep 14 13:12:38.861486 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:38.861448    2557 generic.go:356] "Generic (PLEG): container finished" podID="b70757db-81cb-40e5-9ebb-d98486fb7f97" containerID="c98f27bf340ae742e0ae314a9369ebc3f1d9c92f03da3b4353c5fc86c7c99a68" exitCode=0
Sep 14 13:12:38.862029 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:38.861507    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b" event={"ID":"b70757db-81cb-40e5-9ebb-d98486fb7f97","Type":"ContainerDied","Data":"c98f27bf340ae742e0ae314a9369ebc3f1d9c92f03da3b4353c5fc86c7c99a68"}
Sep 14 13:12:38.862029 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:38.861972    2557 scope.go:117] "RemoveContainer" containerID="c98f27bf340ae742e0ae314a9369ebc3f1d9c92f03da3b4353c5fc86c7c99a68"
Sep 14 13:12:38.863110 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:38.863077    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:12:38.863219 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:38.863153    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:12:38.863219 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:38.863184    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:12:39.873885 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:39.872335    2557 generic.go:356] "Generic (PLEG): container finished" podID="6836e788-4b82-44b3-bfb9-b9718848d0c0" containerID="201ac78560540a8eb5cc24e8764b85d749e538d6453dc9566027aa627091071b" exitCode=0
Sep 14 13:12:39.873885 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:39.872494    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7" event={"ID":"6836e788-4b82-44b3-bfb9-b9718848d0c0","Type":"ContainerDied","Data":"201ac78560540a8eb5cc24e8764b85d749e538d6453dc9566027aa627091071b"}
Sep 14 13:12:39.873885 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:39.873018    2557 scope.go:117] "RemoveContainer" containerID="201ac78560540a8eb5cc24e8764b85d749e538d6453dc9566027aa627091071b"
Sep 14 13:12:39.875489 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:39.875452    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:12:39.875569 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:39.875548    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:12:39.875633 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:39.875593    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:12:39.877686 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:39.877603    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b" event={"ID":"b70757db-81cb-40e5-9ebb-d98486fb7f97","Type":"ContainerStarted","Data":"02d177400785a1665929d1cf8d7c4008c5ee4773c0dd6f96b3afb62085da70a7"}
Sep 14 13:12:40.660998 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:40.660901    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:40.661364 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:40.661275    2557 patch_prober.go:28] interesting pod/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 14 13:12:40.661532 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:40.661511    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b" podUID="b70757db-81cb-40e5-9ebb-d98486fb7f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 14 13:12:40.888263 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:40.888222    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7" event={"ID":"6836e788-4b82-44b3-bfb9-b9718848d0c0","Type":"ContainerStarted","Data":"4c8563a7806796c3bd765c69d83f208a73443d91d8a248c83521094c6739342f"}
Sep 14 13:12:41.660717 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:41.660678    2557 patch_prober.go:28] interesting pod/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 14 13:12:41.661033 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:41.661001    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b" podUID="b70757db-81cb-40e5-9ebb-d98486fb7f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 14 13:12:41.805158 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:41.805119    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:41.805543 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:41.805501    2557 patch_prober.go:28] interesting pod/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 14 13:12:41.805728 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:41.805695    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7" podUID="6836e788-4b82-44b3-bfb9-b9718848d0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 14 13:12:42.665196 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:42.665151    2557 patch_prober.go:28] interesting pod/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 14 13:12:42.665679 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:42.665219    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b" podUID="b70757db-81cb-40e5-9ebb-d98486fb7f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 14 13:12:42.804956 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:42.804917    2557 patch_prober.go:28] interesting pod/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 14 13:12:42.805230 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:42.805199    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7" podUID="6836e788-4b82-44b3-bfb9-b9718848d0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 14 13:12:43.663764 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:43.662100    2557 patch_prober.go:28] interesting pod/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 14 13:12:43.663764 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:43.662166    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b" podUID="b70757db-81cb-40e5-9ebb-d98486fb7f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 14 13:12:43.805449 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:43.805408    2557 patch_prober.go:28] interesting pod/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 14 13:12:43.805957 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:43.805479    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7" podUID="6836e788-4b82-44b3-bfb9-b9718848d0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 14 13:12:44.660524 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:44.660475    2557 patch_prober.go:28] interesting pod/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 14 13:12:44.660763 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:44.660550    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b" podUID="b70757db-81cb-40e5-9ebb-d98486fb7f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 14 13:12:44.804895 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:44.804850    2557 patch_prober.go:28] interesting pod/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 14 13:12:44.805102 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:44.804920    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7" podUID="6836e788-4b82-44b3-bfb9-b9718848d0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 14 13:12:45.724035 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:45.724004    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:45.725078 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:45.725013    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:45.726716 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:45.726696    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:12:45.813566 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:45.813527    2557 patch_prober.go:28] interesting pod/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 14 13:12:45.813879 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:45.813847    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7" podUID="6836e788-4b82-44b3-bfb9-b9718848d0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 14 13:12:45.840669 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:45.840648    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms"
Sep 14 13:12:46.809345 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:46.809308    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:46.809827 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:46.809590    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:46.810330 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:46.810310    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:12:46.848949 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:46.848919    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv"
Sep 14 13:12:49.870189 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:49.870154    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-360610b9-m947f"]
Sep 14 13:12:49.874829 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:49.874800    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-360610b9-m947f"
Sep 14 13:12:49.877379 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:49.877356    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbzp\""
Sep 14 13:12:49.883087 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:49.883057    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-360610b9-m947f"]
Sep 14 13:12:49.926961 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:49.926751    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/643738aa-c57e-43ef-83b3-f0155e96e1de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-360610b9-m947f\" (UID: \"643738aa-c57e-43ef-83b3-f0155e96e1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-360610b9-m947f"
Sep 14 13:12:50.028171 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:50.028131    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/643738aa-c57e-43ef-83b3-f0155e96e1de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-360610b9-m947f\" (UID: \"643738aa-c57e-43ef-83b3-f0155e96e1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-360610b9-m947f"
Sep 14 13:12:50.028780 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:50.028731    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/643738aa-c57e-43ef-83b3-f0155e96e1de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-360610b9-m947f\" (UID: \"643738aa-c57e-43ef-83b3-f0155e96e1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-360610b9-m947f"
Sep 14 13:12:50.209657 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:50.209626    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-360610b9-m947f"
Sep 14 13:12:50.488328 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:50.486889    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-360610b9-m947f"]
Sep 14 13:12:50.489141 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:12:50.489111    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643738aa_c57e_43ef_83b3_f0155e96e1de.slice/crio-2cf91e04cbe78ec8f7814a29b34b30ad2e23529a38e5b31bbe846dca0409bff4 WatchSource:0}: Error finding container 2cf91e04cbe78ec8f7814a29b34b30ad2e23529a38e5b31bbe846dca0409bff4: Status 404 returned error can't find the container with id 2cf91e04cbe78ec8f7814a29b34b30ad2e23529a38e5b31bbe846dca0409bff4
Sep 14 13:12:50.957138 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:50.957105    2557 generic.go:356] "Generic (PLEG): container finished" podID="8110160b-b752-4f51-ab9a-a76e74aa919c" containerID="23b2e7479beed2f36dac93743ea0121538d09128d38d791e4ed5495f5e2942ff" exitCode=0
Sep 14 13:12:50.957593 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:50.957180    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq" event={"ID":"8110160b-b752-4f51-ab9a-a76e74aa919c","Type":"ContainerDied","Data":"23b2e7479beed2f36dac93743ea0121538d09128d38d791e4ed5495f5e2942ff"}
Sep 14 13:12:50.957669 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:50.957626    2557 scope.go:117] "RemoveContainer" containerID="23b2e7479beed2f36dac93743ea0121538d09128d38d791e4ed5495f5e2942ff"
Sep 14 13:12:50.958643 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:50.958616    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-360610b9-m947f" event={"ID":"643738aa-c57e-43ef-83b3-f0155e96e1de","Type":"ContainerStarted","Data":"2cf91e04cbe78ec8f7814a29b34b30ad2e23529a38e5b31bbe846dca0409bff4"}
Sep 14 13:12:50.958959 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:50.958933    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:12:50.959051 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:50.959016    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:12:50.959118 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:50.959066    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:12:51.979522 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:51.979481    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq" event={"ID":"8110160b-b752-4f51-ab9a-a76e74aa919c","Type":"ContainerStarted","Data":"8aaa1b88d5c9b4fc3f93159ef70aca5c18548109a5b6c3777e6c17119457049f"}
Sep 14 13:12:52.764760 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:52.764159    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:52.764760 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:52.764286    2557 patch_prober.go:28] interesting pod/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 14 13:12:52.764760 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:52.764331    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq" podUID="8110160b-b752-4f51-ab9a-a76e74aa919c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 14 13:12:53.762886 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:53.762820    2557 patch_prober.go:28] interesting pod/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 14 13:12:53.763369 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:53.762934    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq" podUID="8110160b-b752-4f51-ab9a-a76e74aa919c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 14 13:12:54.000707 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:54.000663    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-360610b9-m947f" event={"ID":"643738aa-c57e-43ef-83b3-f0155e96e1de","Type":"ContainerStarted","Data":"c43b438a5cda0d7c276d1245762f6d6177b1e3e91fe9e664ba90e6f9115b2e1d"}
Sep 14 13:12:54.020125 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:54.020077    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-360610b9-m947f" podStartSLOduration=2.100329001 podStartE2EDuration="5.020063371s" totalImagesPullingTime="2.91973437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:12:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:12:54.0180352 +0000 UTC m=+892.452710720" watchObservedRunningTime="2026-09-14 13:12:54.020063371 +0000 UTC m=+892.454738890"
Sep 14 13:12:54.763396 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:54.762942    2557 patch_prober.go:28] interesting pod/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 14 13:12:54.763396 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:54.763008    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq" podUID="8110160b-b752-4f51-ab9a-a76e74aa919c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 14 13:12:55.008032 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:55.008001    2557 generic.go:356] "Generic (PLEG): container finished" podID="643738aa-c57e-43ef-83b3-f0155e96e1de" containerID="c43b438a5cda0d7c276d1245762f6d6177b1e3e91fe9e664ba90e6f9115b2e1d" exitCode=0
Sep 14 13:12:55.008235 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:55.008087    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-360610b9-m947f" event={"ID":"643738aa-c57e-43ef-83b3-f0155e96e1de","Type":"ContainerDied","Data":"c43b438a5cda0d7c276d1245762f6d6177b1e3e91fe9e664ba90e6f9115b2e1d"}
Sep 14 13:12:55.763689 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:55.763650    2557 patch_prober.go:28] interesting pod/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 14 13:12:55.764191 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:55.763721    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq" podUID="8110160b-b752-4f51-ab9a-a76e74aa919c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 14 13:12:56.184895 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:56.184873    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-360610b9-m947f"
Sep 14 13:12:56.312954 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:56.312919    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/643738aa-c57e-43ef-83b3-f0155e96e1de-maas-api-service-ca\") pod \"643738aa-c57e-43ef-83b3-f0155e96e1de\" (UID: \"643738aa-c57e-43ef-83b3-f0155e96e1de\") "
Sep 14 13:12:56.313268 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:56.313243    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/643738aa-c57e-43ef-83b3-f0155e96e1de-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "643738aa-c57e-43ef-83b3-f0155e96e1de" (UID: "643738aa-c57e-43ef-83b3-f0155e96e1de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:12:56.413929 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:56.413824    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/643738aa-c57e-43ef-83b3-f0155e96e1de-maas-api-service-ca\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:12:56.763754 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:56.763700    2557 patch_prober.go:28] interesting pod/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 14 13:12:56.764305 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:56.764267    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq" podUID="8110160b-b752-4f51-ab9a-a76e74aa919c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 14 13:12:57.019321 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:57.019238    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-360610b9-m947f"
Sep 14 13:12:57.019458 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:57.019231    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-360610b9-m947f" event={"ID":"643738aa-c57e-43ef-83b3-f0155e96e1de","Type":"ContainerDied","Data":"2cf91e04cbe78ec8f7814a29b34b30ad2e23529a38e5b31bbe846dca0409bff4"}
Sep 14 13:12:57.019458 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:57.019391    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cf91e04cbe78ec8f7814a29b34b30ad2e23529a38e5b31bbe846dca0409bff4"
Sep 14 13:12:57.767600 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:57.767572    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:57.768022 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:57.767866    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:12:57.768589 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:12:57.768570    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:13:00.572878 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:00.572841    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms"]
Sep 14 13:13:00.573295 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:00.573117    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms" podUID="6c02f172-6c1b-4215-8602-2ad5740c0b6c" containerName="payload-processing" containerID="cri-o://a7bd6d0144b910a125132bd2f005de0b087330ae41a530628bdef598a2eb43c4" gracePeriod=30
Sep 14 13:13:00.724302 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:00.724264    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-360610b9-m947f"]
Sep 14 13:13:00.726283 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:00.726257    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-360610b9-m947f"]
Sep 14 13:13:00.850674 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:00.850411    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms"
Sep 14 13:13:00.957580 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:00.957554    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spd27\" (UniqueName: \"kubernetes.io/projected/6c02f172-6c1b-4215-8602-2ad5740c0b6c-kube-api-access-spd27\") pod \"6c02f172-6c1b-4215-8602-2ad5740c0b6c\" (UID: \"6c02f172-6c1b-4215-8602-2ad5740c0b6c\") "
Sep 14 13:13:00.957792 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:00.957616    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c02f172-6c1b-4215-8602-2ad5740c0b6c-plugins-config-volume\") pod \"6c02f172-6c1b-4215-8602-2ad5740c0b6c\" (UID: \"6c02f172-6c1b-4215-8602-2ad5740c0b6c\") "
Sep 14 13:13:00.958021 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:00.957997    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c02f172-6c1b-4215-8602-2ad5740c0b6c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c02f172-6c1b-4215-8602-2ad5740c0b6c" (UID: "6c02f172-6c1b-4215-8602-2ad5740c0b6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:13:00.959573 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:00.959552    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c02f172-6c1b-4215-8602-2ad5740c0b6c-kube-api-access-spd27" (OuterVolumeSpecName: "kube-api-access-spd27") pod "6c02f172-6c1b-4215-8602-2ad5740c0b6c" (UID: "6c02f172-6c1b-4215-8602-2ad5740c0b6c"). InnerVolumeSpecName "kube-api-access-spd27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:13:01.041978 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.041945    2557 generic.go:356] "Generic (PLEG): container finished" podID="6c02f172-6c1b-4215-8602-2ad5740c0b6c" containerID="a7bd6d0144b910a125132bd2f005de0b087330ae41a530628bdef598a2eb43c4" exitCode=0
Sep 14 13:13:01.042163 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.041997    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms"
Sep 14 13:13:01.042163 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.042022    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms" event={"ID":"6c02f172-6c1b-4215-8602-2ad5740c0b6c","Type":"ContainerDied","Data":"a7bd6d0144b910a125132bd2f005de0b087330ae41a530628bdef598a2eb43c4"}
Sep 14 13:13:01.042163 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.042108    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms" event={"ID":"6c02f172-6c1b-4215-8602-2ad5740c0b6c","Type":"ContainerDied","Data":"f27c59115e2de3fbb4df9ecaa7fd46277eb184de364e6be335a33c44d1f9f2b0"}
Sep 14 13:13:01.042163 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.042146    2557 scope.go:117] "RemoveContainer" containerID="a7bd6d0144b910a125132bd2f005de0b087330ae41a530628bdef598a2eb43c4"
Sep 14 13:13:01.052668 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.052641    2557 scope.go:117] "RemoveContainer" containerID="a7bd6d0144b910a125132bd2f005de0b087330ae41a530628bdef598a2eb43c4"
Sep 14 13:13:01.053006 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:13:01.052987    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7bd6d0144b910a125132bd2f005de0b087330ae41a530628bdef598a2eb43c4\": container with ID starting with a7bd6d0144b910a125132bd2f005de0b087330ae41a530628bdef598a2eb43c4 not found: ID does not exist" containerID="a7bd6d0144b910a125132bd2f005de0b087330ae41a530628bdef598a2eb43c4"
Sep 14 13:13:01.053086 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.053014    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7bd6d0144b910a125132bd2f005de0b087330ae41a530628bdef598a2eb43c4"} err="failed to get container status \"a7bd6d0144b910a125132bd2f005de0b087330ae41a530628bdef598a2eb43c4\": rpc error: code = NotFound desc = could not find container \"a7bd6d0144b910a125132bd2f005de0b087330ae41a530628bdef598a2eb43c4\": container with ID starting with a7bd6d0144b910a125132bd2f005de0b087330ae41a530628bdef598a2eb43c4 not found: ID does not exist"
Sep 14 13:13:01.059354 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.059332    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spd27\" (UniqueName: \"kubernetes.io/projected/6c02f172-6c1b-4215-8602-2ad5740c0b6c-kube-api-access-spd27\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:13:01.059354 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.059352    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c02f172-6c1b-4215-8602-2ad5740c0b6c-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:13:01.065155 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.065133    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms"]
Sep 14 13:13:01.069388 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.069368    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-360610b9-dcdbfcb7c-x6rms"]
Sep 14 13:13:01.789050 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.789018    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"]
Sep 14 13:13:01.789594 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.789574    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="643738aa-c57e-43ef-83b3-f0155e96e1de" containerName="revoke-keys"
Sep 14 13:13:01.789674 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.789597    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="643738aa-c57e-43ef-83b3-f0155e96e1de" containerName="revoke-keys"
Sep 14 13:13:01.789674 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.789620    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c02f172-6c1b-4215-8602-2ad5740c0b6c" containerName="payload-processing"
Sep 14 13:13:01.789674 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.789628    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c02f172-6c1b-4215-8602-2ad5740c0b6c" containerName="payload-processing"
Sep 14 13:13:01.789871 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.789753    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="643738aa-c57e-43ef-83b3-f0155e96e1de" containerName="revoke-keys"
Sep 14 13:13:01.789871 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.789770    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c02f172-6c1b-4215-8602-2ad5740c0b6c" containerName="payload-processing"
Sep 14 13:13:01.794246 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.794225    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:01.796958 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.796940    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-ea32f836-gw-openshift-default-dockercfg-b9c5r\""
Sep 14 13:13:01.803916 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.803894    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"]
Sep 14 13:13:01.968183 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.968154    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-workload-certs\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:01.968405 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.968196    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-data\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:01.968405 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.968230    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-token\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:01.968405 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.968261    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2tpd\" (UniqueName: \"kubernetes.io/projected/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-kube-api-access-p2tpd\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:01.968405 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.968292    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-credential-socket\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:01.968405 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.968334    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-workload-socket\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:01.968405 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.968394    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-podinfo\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:01.968719 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.968429    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istiod-ca-cert\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:01.968719 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:01.968460    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-envoy\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:02.050459 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.050380    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="643738aa-c57e-43ef-83b3-f0155e96e1de" path="/var/lib/kubelet/pods/643738aa-c57e-43ef-83b3-f0155e96e1de/volumes"
Sep 14 13:13:02.050929 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.050905    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c02f172-6c1b-4215-8602-2ad5740c0b6c" path="/var/lib/kubelet/pods/6c02f172-6c1b-4215-8602-2ad5740c0b6c/volumes"
Sep 14 13:13:02.052545 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.052522    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwqnf_62d953d6-3b86-4c95-9a47-9579d88993a7/ovn-acl-logging/0.log"
Sep 14 13:13:02.059428 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.059408    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwqnf_62d953d6-3b86-4c95-9a47-9579d88993a7/ovn-acl-logging/0.log"
Sep 14 13:13:02.069840 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.069810    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-workload-certs\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:02.069840 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.069840    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-data\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:02.070006 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.069861    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-token\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:02.070006 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.069894    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2tpd\" (UniqueName: \"kubernetes.io/projected/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-kube-api-access-p2tpd\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:02.070006 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.069936    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-credential-socket\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:02.070006 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.069979    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-workload-socket\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:02.070211 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.070023    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-podinfo\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:02.070211 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.070054    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istiod-ca-cert\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:02.070211 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.070085    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-envoy\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:02.070211 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.070188    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-workload-certs\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:02.070417 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.070251    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-data\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:02.070417 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.070334    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-credential-socket\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:02.070417 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.070343    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-workload-socket\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:02.070771 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.070728    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istiod-ca-cert\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:02.072307 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.072288    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-envoy\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:02.072413 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.072395    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-podinfo\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:02.078251 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.078224    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2tpd\" (UniqueName: \"kubernetes.io/projected/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-kube-api-access-p2tpd\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:02.078681 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.078663    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-token\") pod \"e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") " pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:02.108195 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.108174    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-ea32f836-gw-openshift-default-dockercfg-b9c5r\""
Sep 14 13:13:02.116383 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.116365    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:02.395864 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.395835    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"]
Sep 14 13:13:02.397229 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:13:02.397198    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e96dbf8_347b_4c95_b0bb_f538d17f28b3.slice/crio-4344cecb84f23d566eaafcdf39b8fecc190e2e56b0314a034ae6e98665e5c3c6 WatchSource:0}: Error finding container 4344cecb84f23d566eaafcdf39b8fecc190e2e56b0314a034ae6e98665e5c3c6: Status 404 returned error can't find the container with id 4344cecb84f23d566eaafcdf39b8fecc190e2e56b0314a034ae6e98665e5c3c6
Sep 14 13:13:02.399513 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.399473    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:13:02.399597 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.399545    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:13:02.399597 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:02.399577    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:13:03.055967 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:03.055933    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7" event={"ID":"1e96dbf8-347b-4c95-b0bb-f538d17f28b3","Type":"ContainerStarted","Data":"07f3a7ecd5a43b1a8daa005c2403d6554953b0e0c89755fb113613f97619ff9b"}
Sep 14 13:13:03.056349 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:03.055984    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7" event={"ID":"1e96dbf8-347b-4c95-b0bb-f538d17f28b3","Type":"ContainerStarted","Data":"4344cecb84f23d566eaafcdf39b8fecc190e2e56b0314a034ae6e98665e5c3c6"}
Sep 14 13:13:03.076629 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:03.076570    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7" podStartSLOduration=2.076552422 podStartE2EDuration="2.076552422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:13:03.076217086 +0000 UTC m=+901.510892609" watchObservedRunningTime="2026-09-14 13:13:03.076552422 +0000 UTC m=+901.511227942"
Sep 14 13:13:03.116548 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:03.116526    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:03.118115 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:03.118084    2557 patch_prober.go:28] interesting pod/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 14 13:13:03.118230 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:03.118134    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7" podUID="1e96dbf8-347b-4c95-b0bb-f538d17f28b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 14 13:13:04.120854 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:04.120825    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:05.066432 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:05.066401    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:05.067320 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:05.067296    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:26.174849 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:26.174779    2557 generic.go:356] "Generic (PLEG): container finished" podID="1e96dbf8-347b-4c95-b0bb-f538d17f28b3" containerID="07f3a7ecd5a43b1a8daa005c2403d6554953b0e0c89755fb113613f97619ff9b" exitCode=0
Sep 14 13:13:26.175184 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:26.174855    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7" event={"ID":"1e96dbf8-347b-4c95-b0bb-f538d17f28b3","Type":"ContainerDied","Data":"07f3a7ecd5a43b1a8daa005c2403d6554953b0e0c89755fb113613f97619ff9b"}
Sep 14 13:13:26.175228 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:26.175219    2557 scope.go:117] "RemoveContainer" containerID="07f3a7ecd5a43b1a8daa005c2403d6554953b0e0c89755fb113613f97619ff9b"
Sep 14 13:13:26.176344 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:26.176320    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:13:26.176409 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:26.176385    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:13:26.176449 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:26.176415    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:13:27.183350 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:27.183319    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7" event={"ID":"1e96dbf8-347b-4c95-b0bb-f538d17f28b3","Type":"ContainerStarted","Data":"5f3b8336180b3120ae729193164ac21df979aa217a3371c85fe7feab378a6270"}
Sep 14 13:13:28.117090 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:28.117058    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:28.117354 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:28.117330    2557 patch_prober.go:28] interesting pod/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 14 13:13:28.117454 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:28.117414    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7" podUID="1e96dbf8-347b-4c95-b0bb-f538d17f28b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 14 13:13:29.117114 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:29.117076    2557 patch_prober.go:28] interesting pod/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 14 13:13:29.117513 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:29.117140    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7" podUID="1e96dbf8-347b-4c95-b0bb-f538d17f28b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 14 13:13:29.447010 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:29.446967    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea32f836-tfp8p"]
Sep 14 13:13:29.451447 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:29.451423    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea32f836-tfp8p"
Sep 14 13:13:29.456368 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:29.456137    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbzp\""
Sep 14 13:13:29.461013 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:29.460978    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea32f836-tfp8p"]
Sep 14 13:13:29.521059 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:29.521023    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6db21b-9077-47d0-bc8f-a727baf9ce11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ea32f836-tfp8p\" (UID: \"1b6db21b-9077-47d0-bc8f-a727baf9ce11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea32f836-tfp8p"
Sep 14 13:13:29.622550 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:29.622517    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6db21b-9077-47d0-bc8f-a727baf9ce11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ea32f836-tfp8p\" (UID: \"1b6db21b-9077-47d0-bc8f-a727baf9ce11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea32f836-tfp8p"
Sep 14 13:13:29.623140 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:29.623121    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6db21b-9077-47d0-bc8f-a727baf9ce11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ea32f836-tfp8p\" (UID: \"1b6db21b-9077-47d0-bc8f-a727baf9ce11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea32f836-tfp8p"
Sep 14 13:13:29.773702 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:29.773373    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea32f836-tfp8p"
Sep 14 13:13:30.093882 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:30.093780    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea32f836-tfp8p"]
Sep 14 13:13:30.117578 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:30.117549    2557 patch_prober.go:28] interesting pod/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 14 13:13:30.117883 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:30.117610    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7" podUID="1e96dbf8-347b-4c95-b0bb-f538d17f28b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 14 13:13:30.200157 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:30.200123    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea32f836-tfp8p" event={"ID":"1b6db21b-9077-47d0-bc8f-a727baf9ce11","Type":"ContainerStarted","Data":"27e40e3be04291d7bd571d8de12c5224e1351507f1201df69abcf27573bfacb1"}
Sep 14 13:13:31.116795 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:31.116762    2557 patch_prober.go:28] interesting pod/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 14 13:13:31.116951 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:31.116827    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7" podUID="1e96dbf8-347b-4c95-b0bb-f538d17f28b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 14 13:13:31.207032 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:31.207000    2557 generic.go:356] "Generic (PLEG): container finished" podID="1b6db21b-9077-47d0-bc8f-a727baf9ce11" containerID="602ccc6b856446db23203a16b67d2b5562a689eecc06ca5f384623c5db980b8d" exitCode=0
Sep 14 13:13:31.207417 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:31.207086    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea32f836-tfp8p" event={"ID":"1b6db21b-9077-47d0-bc8f-a727baf9ce11","Type":"ContainerDied","Data":"602ccc6b856446db23203a16b67d2b5562a689eecc06ca5f384623c5db980b8d"}
Sep 14 13:13:32.116788 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:32.116728    2557 patch_prober.go:28] interesting pod/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 14 13:13:32.116961 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:32.116805    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7" podUID="1e96dbf8-347b-4c95-b0bb-f538d17f28b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 14 13:13:32.402906 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:32.402784    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea32f836-tfp8p"
Sep 14 13:13:32.566763 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:32.565728    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6db21b-9077-47d0-bc8f-a727baf9ce11-maas-api-service-ca\") pod \"1b6db21b-9077-47d0-bc8f-a727baf9ce11\" (UID: \"1b6db21b-9077-47d0-bc8f-a727baf9ce11\") "
Sep 14 13:13:32.566763 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:32.566096    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b6db21b-9077-47d0-bc8f-a727baf9ce11-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b6db21b-9077-47d0-bc8f-a727baf9ce11" (UID: "1b6db21b-9077-47d0-bc8f-a727baf9ce11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:13:32.566763 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:32.566231    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6db21b-9077-47d0-bc8f-a727baf9ce11-maas-api-service-ca\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:13:33.121173 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:33.121143    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:33.121444 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:33.121427    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:33.122272 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:33.122250    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:33.217792 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:33.217730    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea32f836-tfp8p"
Sep 14 13:13:33.217953 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:33.217883    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea32f836-tfp8p" event={"ID":"1b6db21b-9077-47d0-bc8f-a727baf9ce11","Type":"ContainerDied","Data":"27e40e3be04291d7bd571d8de12c5224e1351507f1201df69abcf27573bfacb1"}
Sep 14 13:13:33.217953 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:33.217934    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27e40e3be04291d7bd571d8de12c5224e1351507f1201df69abcf27573bfacb1"
Sep 14 13:13:36.515242 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:36.515204    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea32f836-tfp8p"]
Sep 14 13:13:36.518697 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:36.518669    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea32f836-tfp8p"]
Sep 14 13:13:36.643169 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:36.643138    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"]
Sep 14 13:13:36.643378 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:36.643348    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7" podUID="1e96dbf8-347b-4c95-b0bb-f538d17f28b3" containerName="istio-proxy" containerID="cri-o://5f3b8336180b3120ae729193164ac21df979aa217a3371c85fe7feab378a6270" gracePeriod=30
Sep 14 13:13:38.049552 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:38.049516    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b6db21b-9077-47d0-bc8f-a727baf9ce11" path="/var/lib/kubelet/pods/1b6db21b-9077-47d0-bc8f-a727baf9ce11/volumes"
Sep 14 13:13:41.904113 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:41.904057    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:42.053860 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.053828    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-podinfo\") pod \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") "
Sep 14 13:13:42.054018 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.053868    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2tpd\" (UniqueName: \"kubernetes.io/projected/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-kube-api-access-p2tpd\") pod \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") "
Sep 14 13:13:42.054018 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.053900    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-workload-socket\") pod \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") "
Sep 14 13:13:42.054018 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.053977    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-data\") pod \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") "
Sep 14 13:13:42.054018 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.053999    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-token\") pod \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") "
Sep 14 13:13:42.054298 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.054042    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-envoy\") pod \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") "
Sep 14 13:13:42.054298 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.054082    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-workload-certs\") pod \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") "
Sep 14 13:13:42.054298 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.054115    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-credential-socket\") pod \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") "
Sep 14 13:13:42.054298 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.054167    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istiod-ca-cert\") pod \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\" (UID: \"1e96dbf8-347b-4c95-b0bb-f538d17f28b3\") "
Sep 14 13:13:42.054298 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.054232    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1e96dbf8-347b-4c95-b0bb-f538d17f28b3" (UID: "1e96dbf8-347b-4c95-b0bb-f538d17f28b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:13:42.054551 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.054272    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-data" (OuterVolumeSpecName: "istio-data") pod "1e96dbf8-347b-4c95-b0bb-f538d17f28b3" (UID: "1e96dbf8-347b-4c95-b0bb-f538d17f28b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:13:42.054551 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.054328    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1e96dbf8-347b-4c95-b0bb-f538d17f28b3" (UID: "1e96dbf8-347b-4c95-b0bb-f538d17f28b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:13:42.054551 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.054397    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1e96dbf8-347b-4c95-b0bb-f538d17f28b3" (UID: "1e96dbf8-347b-4c95-b0bb-f538d17f28b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:13:42.054551 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.054512    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-workload-socket\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:13:42.054551 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.054530    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-data\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:13:42.054551 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.054544    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-workload-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:13:42.054816 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.054556    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-credential-socket\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:13:42.054816 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.054582    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1e96dbf8-347b-4c95-b0bb-f538d17f28b3" (UID: "1e96dbf8-347b-4c95-b0bb-f538d17f28b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:13:42.056042 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.056013    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-kube-api-access-p2tpd" (OuterVolumeSpecName: "kube-api-access-p2tpd") pod "1e96dbf8-347b-4c95-b0bb-f538d17f28b3" (UID: "1e96dbf8-347b-4c95-b0bb-f538d17f28b3"). InnerVolumeSpecName "kube-api-access-p2tpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:13:42.056283 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.056263    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1e96dbf8-347b-4c95-b0bb-f538d17f28b3" (UID: "1e96dbf8-347b-4c95-b0bb-f538d17f28b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 13:13:42.056543 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.056513    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1e96dbf8-347b-4c95-b0bb-f538d17f28b3" (UID: "1e96dbf8-347b-4c95-b0bb-f538d17f28b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:13:42.056543 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.056525    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "1e96dbf8-347b-4c95-b0bb-f538d17f28b3" (UID: "1e96dbf8-347b-4c95-b0bb-f538d17f28b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:13:42.087081 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.087058    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm"]
Sep 14 13:13:42.087428 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.087416    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b6db21b-9077-47d0-bc8f-a727baf9ce11" containerName="revoke-keys"
Sep 14 13:13:42.087470 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.087430    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b6db21b-9077-47d0-bc8f-a727baf9ce11" containerName="revoke-keys"
Sep 14 13:13:42.087470 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.087449    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e96dbf8-347b-4c95-b0bb-f538d17f28b3" containerName="istio-proxy"
Sep 14 13:13:42.087470 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.087454    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e96dbf8-347b-4c95-b0bb-f538d17f28b3" containerName="istio-proxy"
Sep 14 13:13:42.087470 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.087461    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e96dbf8-347b-4c95-b0bb-f538d17f28b3" containerName="istio-proxy"
Sep 14 13:13:42.087470 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.087467    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e96dbf8-347b-4c95-b0bb-f538d17f28b3" containerName="istio-proxy"
Sep 14 13:13:42.087623 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.087530    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e96dbf8-347b-4c95-b0bb-f538d17f28b3" containerName="istio-proxy"
Sep 14 13:13:42.087623 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.087535    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b6db21b-9077-47d0-bc8f-a727baf9ce11" containerName="revoke-keys"
Sep 14 13:13:42.087623 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.087543    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e96dbf8-347b-4c95-b0bb-f538d17f28b3" containerName="istio-proxy"
Sep 14 13:13:42.090763 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.090730    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm"
Sep 14 13:13:42.093344 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.093328    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-dd15e28a-dockercfg-dzf95\""
Sep 14 13:13:42.093419 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.093369    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-dd15e28a\""
Sep 14 13:13:42.100697 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.100672    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm"]
Sep 14 13:13:42.117700 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.117678    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt"]
Sep 14 13:13:42.121099 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.121084    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt"
Sep 14 13:13:42.129548 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.129526    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt"]
Sep 14 13:13:42.155725 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.155667    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-envoy\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:13:42.155725 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.155687    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istiod-ca-cert\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:13:42.155725 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.155695    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-podinfo\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:13:42.155725 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.155704    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2tpd\" (UniqueName: \"kubernetes.io/projected/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-kube-api-access-p2tpd\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:13:42.155725 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.155712    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e96dbf8-347b-4c95-b0bb-f538d17f28b3-istio-token\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:13:42.256204 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.256175    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e58a84f-8110-4719-a251-47b1dff522e4-plugins-config-volume\") pod \"payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt\" (UID: \"6e58a84f-8110-4719-a251-47b1dff522e4\") " pod="openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt"
Sep 14 13:13:42.256385 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.256260    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p848r\" (UniqueName: \"kubernetes.io/projected/6e58a84f-8110-4719-a251-47b1dff522e4-kube-api-access-p848r\") pod \"payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt\" (UID: \"6e58a84f-8110-4719-a251-47b1dff522e4\") " pod="openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt"
Sep 14 13:13:42.256385 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.256344    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv99s\" (UniqueName: \"kubernetes.io/projected/478bdff7-7449-47ea-8983-3be142ce4028-kube-api-access-wv99s\") pod \"payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm\" (UID: \"478bdff7-7449-47ea-8983-3be142ce4028\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm"
Sep 14 13:13:42.256495 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.256386    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478bdff7-7449-47ea-8983-3be142ce4028-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm\" (UID: \"478bdff7-7449-47ea-8983-3be142ce4028\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm"
Sep 14 13:13:42.272895 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.272860    2557 generic.go:356] "Generic (PLEG): container finished" podID="1e96dbf8-347b-4c95-b0bb-f538d17f28b3" containerID="5f3b8336180b3120ae729193164ac21df979aa217a3371c85fe7feab378a6270" exitCode=0
Sep 14 13:13:42.273054 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.272917    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"
Sep 14 13:13:42.273054 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.272940    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7" event={"ID":"1e96dbf8-347b-4c95-b0bb-f538d17f28b3","Type":"ContainerDied","Data":"5f3b8336180b3120ae729193164ac21df979aa217a3371c85fe7feab378a6270"}
Sep 14 13:13:42.273054 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.273002    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7" event={"ID":"1e96dbf8-347b-4c95-b0bb-f538d17f28b3","Type":"ContainerDied","Data":"4344cecb84f23d566eaafcdf39b8fecc190e2e56b0314a034ae6e98665e5c3c6"}
Sep 14 13:13:42.273054 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.273027    2557 scope.go:117] "RemoveContainer" containerID="5f3b8336180b3120ae729193164ac21df979aa217a3371c85fe7feab378a6270"
Sep 14 13:13:42.283746 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.283717    2557 scope.go:117] "RemoveContainer" containerID="07f3a7ecd5a43b1a8daa005c2403d6554953b0e0c89755fb113613f97619ff9b"
Sep 14 13:13:42.291801 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.291780    2557 scope.go:117] "RemoveContainer" containerID="5f3b8336180b3120ae729193164ac21df979aa217a3371c85fe7feab378a6270"
Sep 14 13:13:42.292040 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:13:42.292024    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f3b8336180b3120ae729193164ac21df979aa217a3371c85fe7feab378a6270\": container with ID starting with 5f3b8336180b3120ae729193164ac21df979aa217a3371c85fe7feab378a6270 not found: ID does not exist" containerID="5f3b8336180b3120ae729193164ac21df979aa217a3371c85fe7feab378a6270"
Sep 14 13:13:42.292112 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.292047    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f3b8336180b3120ae729193164ac21df979aa217a3371c85fe7feab378a6270"} err="failed to get container status \"5f3b8336180b3120ae729193164ac21df979aa217a3371c85fe7feab378a6270\": rpc error: code = NotFound desc = could not find container \"5f3b8336180b3120ae729193164ac21df979aa217a3371c85fe7feab378a6270\": container with ID starting with 5f3b8336180b3120ae729193164ac21df979aa217a3371c85fe7feab378a6270 not found: ID does not exist"
Sep 14 13:13:42.292112 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.292065    2557 scope.go:117] "RemoveContainer" containerID="07f3a7ecd5a43b1a8daa005c2403d6554953b0e0c89755fb113613f97619ff9b"
Sep 14 13:13:42.292455 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:13:42.292419    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07f3a7ecd5a43b1a8daa005c2403d6554953b0e0c89755fb113613f97619ff9b\": container with ID starting with 07f3a7ecd5a43b1a8daa005c2403d6554953b0e0c89755fb113613f97619ff9b not found: ID does not exist" containerID="07f3a7ecd5a43b1a8daa005c2403d6554953b0e0c89755fb113613f97619ff9b"
Sep 14 13:13:42.292647 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.292621    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07f3a7ecd5a43b1a8daa005c2403d6554953b0e0c89755fb113613f97619ff9b"} err="failed to get container status \"07f3a7ecd5a43b1a8daa005c2403d6554953b0e0c89755fb113613f97619ff9b\": rpc error: code = NotFound desc = could not find container \"07f3a7ecd5a43b1a8daa005c2403d6554953b0e0c89755fb113613f97619ff9b\": container with ID starting with 07f3a7ecd5a43b1a8daa005c2403d6554953b0e0c89755fb113613f97619ff9b not found: ID does not exist"
Sep 14 13:13:42.294839 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.294822    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"]
Sep 14 13:13:42.298642 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.298624    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-ea32f836-gw-openshift-default-747c4f4584-6m8n7"]
Sep 14 13:13:42.357523 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.357499    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv99s\" (UniqueName: \"kubernetes.io/projected/478bdff7-7449-47ea-8983-3be142ce4028-kube-api-access-wv99s\") pod \"payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm\" (UID: \"478bdff7-7449-47ea-8983-3be142ce4028\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm"
Sep 14 13:13:42.357635 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.357535    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478bdff7-7449-47ea-8983-3be142ce4028-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm\" (UID: \"478bdff7-7449-47ea-8983-3be142ce4028\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm"
Sep 14 13:13:42.357635 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.357599    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e58a84f-8110-4719-a251-47b1dff522e4-plugins-config-volume\") pod \"payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt\" (UID: \"6e58a84f-8110-4719-a251-47b1dff522e4\") " pod="openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt"
Sep 14 13:13:42.357635 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.357628    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p848r\" (UniqueName: \"kubernetes.io/projected/6e58a84f-8110-4719-a251-47b1dff522e4-kube-api-access-p848r\") pod \"payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt\" (UID: \"6e58a84f-8110-4719-a251-47b1dff522e4\") " pod="openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt"
Sep 14 13:13:42.358334 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.358316    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e58a84f-8110-4719-a251-47b1dff522e4-plugins-config-volume\") pod \"payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt\" (UID: \"6e58a84f-8110-4719-a251-47b1dff522e4\") " pod="openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt"
Sep 14 13:13:42.358441 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.358420    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478bdff7-7449-47ea-8983-3be142ce4028-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm\" (UID: \"478bdff7-7449-47ea-8983-3be142ce4028\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm"
Sep 14 13:13:42.366780 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.366730    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p848r\" (UniqueName: \"kubernetes.io/projected/6e58a84f-8110-4719-a251-47b1dff522e4-kube-api-access-p848r\") pod \"payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt\" (UID: \"6e58a84f-8110-4719-a251-47b1dff522e4\") " pod="openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt"
Sep 14 13:13:42.366867 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.366840    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv99s\" (UniqueName: \"kubernetes.io/projected/478bdff7-7449-47ea-8983-3be142ce4028-kube-api-access-wv99s\") pod \"payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm\" (UID: \"478bdff7-7449-47ea-8983-3be142ce4028\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm"
Sep 14 13:13:42.402305 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.402276    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm"
Sep 14 13:13:42.432594 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.432574    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt"
Sep 14 13:13:42.695712 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.695688    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm"]
Sep 14 13:13:42.698012 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:13:42.697987    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod478bdff7_7449_47ea_8983_3be142ce4028.slice/crio-9ea36e3c7b3b1835570a029b96b619bace20e45db084009dad918911bed2383e WatchSource:0}: Error finding container 9ea36e3c7b3b1835570a029b96b619bace20e45db084009dad918911bed2383e: Status 404 returned error can't find the container with id 9ea36e3c7b3b1835570a029b96b619bace20e45db084009dad918911bed2383e
Sep 14 13:13:42.747486 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:42.747462    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt"]
Sep 14 13:13:42.748054 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:13:42.748034    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e58a84f_8110_4719_a251_47b1dff522e4.slice/crio-b12bbd3f6ecc1b26fc3a66eac32d8839a5b214ac9058a4be045288119b9c5fb8 WatchSource:0}: Error finding container b12bbd3f6ecc1b26fc3a66eac32d8839a5b214ac9058a4be045288119b9c5fb8: Status 404 returned error can't find the container with id b12bbd3f6ecc1b26fc3a66eac32d8839a5b214ac9058a4be045288119b9c5fb8
Sep 14 13:13:43.281956 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:43.281917    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm" event={"ID":"478bdff7-7449-47ea-8983-3be142ce4028","Type":"ContainerStarted","Data":"c2ae2e3ed70950c49bd782bba309053877cb4d6c2c65de5ed777fd3934a8a488"}
Sep 14 13:13:43.282413 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:43.282015    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm" event={"ID":"478bdff7-7449-47ea-8983-3be142ce4028","Type":"ContainerStarted","Data":"9ea36e3c7b3b1835570a029b96b619bace20e45db084009dad918911bed2383e"}
Sep 14 13:13:43.282413 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:43.282049    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm"
Sep 14 13:13:43.283490 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:43.283467    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt" event={"ID":"6e58a84f-8110-4719-a251-47b1dff522e4","Type":"ContainerStarted","Data":"26da2311d92072bb7d41c8023a324af03eb40b1fda4a1917f77c3257b0175fb9"}
Sep 14 13:13:43.283602 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:43.283511    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt" event={"ID":"6e58a84f-8110-4719-a251-47b1dff522e4","Type":"ContainerStarted","Data":"b12bbd3f6ecc1b26fc3a66eac32d8839a5b214ac9058a4be045288119b9c5fb8"}
Sep 14 13:13:43.283662 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:43.283605    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt"
Sep 14 13:13:43.298850 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:43.298803    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm" podStartSLOduration=1.298788004 podStartE2EDuration="1.298788004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:13:43.298231226 +0000 UTC m=+941.732906748" watchObservedRunningTime="2026-09-14 13:13:43.298788004 +0000 UTC m=+941.733463524"
Sep 14 13:13:43.318160 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:43.318115    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt" podStartSLOduration=1.318100529 podStartE2EDuration="1.318100529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:13:43.315826111 +0000 UTC m=+941.750501633" watchObservedRunningTime="2026-09-14 13:13:43.318100529 +0000 UTC m=+941.752776047"
Sep 14 13:13:44.050796 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:44.050731    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e96dbf8-347b-4c95-b0bb-f538d17f28b3" path="/var/lib/kubelet/pods/1e96dbf8-347b-4c95-b0bb-f538d17f28b3/volumes"
Sep 14 13:13:48.203514 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.203461    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"]
Sep 14 13:13:48.207499 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.207474    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:48.207959 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.207935    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-254hf\" (UniqueName: \"kubernetes.io/projected/bad29697-772e-458b-a7cf-38e441ccf37f-kube-api-access-254hf\") pod \"autores-model-c14a5e-kserve-58dfddb4bc-btfkd\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") " pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:48.208040 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.207983    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-tmp-dir\") pod \"autores-model-c14a5e-kserve-58dfddb4bc-btfkd\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") " pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:48.208040 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.208008    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-home\") pod \"autores-model-c14a5e-kserve-58dfddb4bc-btfkd\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") " pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:48.208116 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.208056    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bad29697-772e-458b-a7cf-38e441ccf37f-tls-certs\") pod \"autores-model-c14a5e-kserve-58dfddb4bc-btfkd\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") " pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:48.208149 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.208128    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-model-cache\") pod \"autores-model-c14a5e-kserve-58dfddb4bc-btfkd\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") " pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:48.208203 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.208183    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-dshm\") pod \"autores-model-c14a5e-kserve-58dfddb4bc-btfkd\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") " pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:48.211615 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.211587    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c14a5e\"/\"autores-model-c14a5e-kserve-self-signed-certs\""
Sep 14 13:13:48.211615 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.211595    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c14a5e\"/\"kube-root-ca.crt\""
Sep 14 13:13:48.211772 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.211602    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c14a5e\"/\"openshift-service-ca.crt\""
Sep 14 13:13:48.211772 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.211654    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c14a5e\"/\"default-dockercfg-wv6jn\""
Sep 14 13:13:48.217107 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.217086    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"]
Sep 14 13:13:48.308938 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.308904    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bad29697-772e-458b-a7cf-38e441ccf37f-tls-certs\") pod \"autores-model-c14a5e-kserve-58dfddb4bc-btfkd\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") " pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:48.309107 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.308990    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-model-cache\") pod \"autores-model-c14a5e-kserve-58dfddb4bc-btfkd\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") " pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:48.309107 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.309025    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-dshm\") pod \"autores-model-c14a5e-kserve-58dfddb4bc-btfkd\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") " pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:48.309107 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.309070    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-254hf\" (UniqueName: \"kubernetes.io/projected/bad29697-772e-458b-a7cf-38e441ccf37f-kube-api-access-254hf\") pod \"autores-model-c14a5e-kserve-58dfddb4bc-btfkd\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") " pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:48.309262 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.309108    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-tmp-dir\") pod \"autores-model-c14a5e-kserve-58dfddb4bc-btfkd\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") " pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:48.309262 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.309136    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-home\") pod \"autores-model-c14a5e-kserve-58dfddb4bc-btfkd\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") " pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:48.309447 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.309424    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-model-cache\") pod \"autores-model-c14a5e-kserve-58dfddb4bc-btfkd\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") " pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:48.309610 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.309508    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-tmp-dir\") pod \"autores-model-c14a5e-kserve-58dfddb4bc-btfkd\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") " pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:48.309693 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.309537    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-home\") pod \"autores-model-c14a5e-kserve-58dfddb4bc-btfkd\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") " pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:48.311341 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.311314    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-dshm\") pod \"autores-model-c14a5e-kserve-58dfddb4bc-btfkd\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") " pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:48.311490 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.311473    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bad29697-772e-458b-a7cf-38e441ccf37f-tls-certs\") pod \"autores-model-c14a5e-kserve-58dfddb4bc-btfkd\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") " pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:48.319904 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.319885    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-254hf\" (UniqueName: \"kubernetes.io/projected/bad29697-772e-458b-a7cf-38e441ccf37f-kube-api-access-254hf\") pod \"autores-model-c14a5e-kserve-58dfddb4bc-btfkd\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") " pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:48.516971 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.516888    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:48.792230 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:48.792203    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"]
Sep 14 13:13:48.794475 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:13:48.794449    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbad29697_772e_458b_a7cf_38e441ccf37f.slice/crio-5e6d2a686d9f81630f3d5b15b44921e23ce33a4ac0767e606fe55cf9e0822289 WatchSource:0}: Error finding container 5e6d2a686d9f81630f3d5b15b44921e23ce33a4ac0767e606fe55cf9e0822289: Status 404 returned error can't find the container with id 5e6d2a686d9f81630f3d5b15b44921e23ce33a4ac0767e606fe55cf9e0822289
Sep 14 13:13:49.317659 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:49.317622    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd" event={"ID":"bad29697-772e-458b-a7cf-38e441ccf37f","Type":"ContainerStarted","Data":"af2c5145cdf05d85d1a8b283795b0f76d2c8f4f2a91d93a19a5fb3902b02f566"}
Sep 14 13:13:49.318040 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:49.317665    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd" event={"ID":"bad29697-772e-458b-a7cf-38e441ccf37f","Type":"ContainerStarted","Data":"5e6d2a686d9f81630f3d5b15b44921e23ce33a4ac0767e606fe55cf9e0822289"}
Sep 14 13:13:49.336567 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:49.336521    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd" podStartSLOduration=1.336507751 podStartE2EDuration="1.336507751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:13:49.335569008 +0000 UTC m=+947.770244538" watchObservedRunningTime="2026-09-14 13:13:49.336507751 +0000 UTC m=+947.771183301"
Sep 14 13:13:54.291270 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:54.291242    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt"
Sep 14 13:13:54.291786 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:54.291310    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm"
Sep 14 13:13:58.517862 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:58.517802    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:58.518278 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:58.517882    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:58.530825 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:58.530802    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:13:59.380633 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:13:59.380606    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:14:05.726691 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:05.726655    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd15e28a-sc9jm"]
Sep 14 13:14:05.731444 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:05.731426    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd15e28a-sc9jm"
Sep 14 13:14:05.735460 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:05.735441    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbzp\""
Sep 14 13:14:05.739537 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:05.739517    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd15e28a-sc9jm"]
Sep 14 13:14:05.762170 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:05.762142    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c539994b-14df-493a-b009-32beb38a4e30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dd15e28a-sc9jm\" (UID: \"c539994b-14df-493a-b009-32beb38a4e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd15e28a-sc9jm"
Sep 14 13:14:05.862704 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:05.862680    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c539994b-14df-493a-b009-32beb38a4e30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dd15e28a-sc9jm\" (UID: \"c539994b-14df-493a-b009-32beb38a4e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd15e28a-sc9jm"
Sep 14 13:14:05.863335 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:05.863316    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c539994b-14df-493a-b009-32beb38a4e30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dd15e28a-sc9jm\" (UID: \"c539994b-14df-493a-b009-32beb38a4e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd15e28a-sc9jm"
Sep 14 13:14:06.044580 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:06.044510    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd15e28a-sc9jm"
Sep 14 13:14:06.316877 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:06.316855    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd15e28a-sc9jm"]
Sep 14 13:14:06.318792 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:14:06.318760    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc539994b_14df_493a_b009_32beb38a4e30.slice/crio-001e38448aeb86b8e4c8cd80985271d4c8130094287915ae7cd6d5a47d1ef9fc WatchSource:0}: Error finding container 001e38448aeb86b8e4c8cd80985271d4c8130094287915ae7cd6d5a47d1ef9fc: Status 404 returned error can't find the container with id 001e38448aeb86b8e4c8cd80985271d4c8130094287915ae7cd6d5a47d1ef9fc
Sep 14 13:14:06.411317 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:06.411291    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd15e28a-sc9jm" event={"ID":"c539994b-14df-493a-b009-32beb38a4e30","Type":"ContainerStarted","Data":"001e38448aeb86b8e4c8cd80985271d4c8130094287915ae7cd6d5a47d1ef9fc"}
Sep 14 13:14:07.418419 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:07.418347    2557 generic.go:356] "Generic (PLEG): container finished" podID="c539994b-14df-493a-b009-32beb38a4e30" containerID="e5baac40f683d0f35037d9d339a978ee73a349c22b1f62a1ae0d84226a399fec" exitCode=0
Sep 14 13:14:07.418838 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:07.418429    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd15e28a-sc9jm" event={"ID":"c539994b-14df-493a-b009-32beb38a4e30","Type":"ContainerDied","Data":"e5baac40f683d0f35037d9d339a978ee73a349c22b1f62a1ae0d84226a399fec"}
Sep 14 13:14:08.567160 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:08.567135    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd15e28a-sc9jm"
Sep 14 13:14:08.689605 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:08.689585    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c539994b-14df-493a-b009-32beb38a4e30-maas-api-service-ca\") pod \"c539994b-14df-493a-b009-32beb38a4e30\" (UID: \"c539994b-14df-493a-b009-32beb38a4e30\") "
Sep 14 13:14:08.690003 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:08.689981    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c539994b-14df-493a-b009-32beb38a4e30-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c539994b-14df-493a-b009-32beb38a4e30" (UID: "c539994b-14df-493a-b009-32beb38a4e30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:14:08.790754 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:08.790708    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c539994b-14df-493a-b009-32beb38a4e30-maas-api-service-ca\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:09.430128 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:09.430103    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd15e28a-sc9jm"
Sep 14 13:14:09.430360 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:09.430138    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd15e28a-sc9jm" event={"ID":"c539994b-14df-493a-b009-32beb38a4e30","Type":"ContainerDied","Data":"001e38448aeb86b8e4c8cd80985271d4c8130094287915ae7cd6d5a47d1ef9fc"}
Sep 14 13:14:09.430360 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:09.430182    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="001e38448aeb86b8e4c8cd80985271d4c8130094287915ae7cd6d5a47d1ef9fc"
Sep 14 13:14:12.495403 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:12.495364    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt"]
Sep 14 13:14:12.495882 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:12.495616    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt" podUID="6e58a84f-8110-4719-a251-47b1dff522e4" containerName="payload-processing" containerID="cri-o://26da2311d92072bb7d41c8023a324af03eb40b1fda4a1917f77c3257b0175fb9" gracePeriod=30
Sep 14 13:14:12.503152 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:12.503123    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm"]
Sep 14 13:14:12.503379 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:12.503349    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm" podUID="478bdff7-7449-47ea-8983-3be142ce4028" containerName="payload-pre-processing" containerID="cri-o://c2ae2e3ed70950c49bd782bba309053877cb4d6c2c65de5ed777fd3934a8a488" gracePeriod=30
Sep 14 13:14:12.730470 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:12.730431    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd15e28a-sc9jm"]
Sep 14 13:14:12.734989 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:12.734958    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd15e28a-sc9jm"]
Sep 14 13:14:12.794656 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:12.794635    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt"
Sep 14 13:14:12.797836 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:12.797811    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm"
Sep 14 13:14:12.930758 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:12.930706    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478bdff7-7449-47ea-8983-3be142ce4028-plugins-config-volume\") pod \"478bdff7-7449-47ea-8983-3be142ce4028\" (UID: \"478bdff7-7449-47ea-8983-3be142ce4028\") "
Sep 14 13:14:12.930917 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:12.930781    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv99s\" (UniqueName: \"kubernetes.io/projected/478bdff7-7449-47ea-8983-3be142ce4028-kube-api-access-wv99s\") pod \"478bdff7-7449-47ea-8983-3be142ce4028\" (UID: \"478bdff7-7449-47ea-8983-3be142ce4028\") "
Sep 14 13:14:12.930917 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:12.930817    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e58a84f-8110-4719-a251-47b1dff522e4-plugins-config-volume\") pod \"6e58a84f-8110-4719-a251-47b1dff522e4\" (UID: \"6e58a84f-8110-4719-a251-47b1dff522e4\") "
Sep 14 13:14:12.930917 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:12.930842    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p848r\" (UniqueName: \"kubernetes.io/projected/6e58a84f-8110-4719-a251-47b1dff522e4-kube-api-access-p848r\") pod \"6e58a84f-8110-4719-a251-47b1dff522e4\" (UID: \"6e58a84f-8110-4719-a251-47b1dff522e4\") "
Sep 14 13:14:12.931157 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:12.931126    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/478bdff7-7449-47ea-8983-3be142ce4028-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "478bdff7-7449-47ea-8983-3be142ce4028" (UID: "478bdff7-7449-47ea-8983-3be142ce4028"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:14:12.931226 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:12.931149    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e58a84f-8110-4719-a251-47b1dff522e4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e58a84f-8110-4719-a251-47b1dff522e4" (UID: "6e58a84f-8110-4719-a251-47b1dff522e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:14:12.932931 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:12.932895    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/478bdff7-7449-47ea-8983-3be142ce4028-kube-api-access-wv99s" (OuterVolumeSpecName: "kube-api-access-wv99s") pod "478bdff7-7449-47ea-8983-3be142ce4028" (UID: "478bdff7-7449-47ea-8983-3be142ce4028"). InnerVolumeSpecName "kube-api-access-wv99s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:14:12.932931 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:12.932916    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e58a84f-8110-4719-a251-47b1dff522e4-kube-api-access-p848r" (OuterVolumeSpecName: "kube-api-access-p848r") pod "6e58a84f-8110-4719-a251-47b1dff522e4" (UID: "6e58a84f-8110-4719-a251-47b1dff522e4"). InnerVolumeSpecName "kube-api-access-p848r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:14:13.032352 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:13.032275    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv99s\" (UniqueName: \"kubernetes.io/projected/478bdff7-7449-47ea-8983-3be142ce4028-kube-api-access-wv99s\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:13.032352 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:13.032302    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e58a84f-8110-4719-a251-47b1dff522e4-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:13.032352 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:13.032315    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p848r\" (UniqueName: \"kubernetes.io/projected/6e58a84f-8110-4719-a251-47b1dff522e4-kube-api-access-p848r\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:13.032352 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:13.032328    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478bdff7-7449-47ea-8983-3be142ce4028-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:13.453493 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:13.453461    2557 generic.go:356] "Generic (PLEG): container finished" podID="6e58a84f-8110-4719-a251-47b1dff522e4" containerID="26da2311d92072bb7d41c8023a324af03eb40b1fda4a1917f77c3257b0175fb9" exitCode=0
Sep 14 13:14:13.453660 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:13.453514    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt"
Sep 14 13:14:13.453660 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:13.453538    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt" event={"ID":"6e58a84f-8110-4719-a251-47b1dff522e4","Type":"ContainerDied","Data":"26da2311d92072bb7d41c8023a324af03eb40b1fda4a1917f77c3257b0175fb9"}
Sep 14 13:14:13.453660 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:13.453590    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt" event={"ID":"6e58a84f-8110-4719-a251-47b1dff522e4","Type":"ContainerDied","Data":"b12bbd3f6ecc1b26fc3a66eac32d8839a5b214ac9058a4be045288119b9c5fb8"}
Sep 14 13:14:13.453660 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:13.453607    2557 scope.go:117] "RemoveContainer" containerID="26da2311d92072bb7d41c8023a324af03eb40b1fda4a1917f77c3257b0175fb9"
Sep 14 13:14:13.455109 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:13.455084    2557 generic.go:356] "Generic (PLEG): container finished" podID="478bdff7-7449-47ea-8983-3be142ce4028" containerID="c2ae2e3ed70950c49bd782bba309053877cb4d6c2c65de5ed777fd3934a8a488" exitCode=0
Sep 14 13:14:13.455218 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:13.455162    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm"
Sep 14 13:14:13.455218 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:13.455170    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm" event={"ID":"478bdff7-7449-47ea-8983-3be142ce4028","Type":"ContainerDied","Data":"c2ae2e3ed70950c49bd782bba309053877cb4d6c2c65de5ed777fd3934a8a488"}
Sep 14 13:14:13.455296 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:13.455270    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm" event={"ID":"478bdff7-7449-47ea-8983-3be142ce4028","Type":"ContainerDied","Data":"9ea36e3c7b3b1835570a029b96b619bace20e45db084009dad918911bed2383e"}
Sep 14 13:14:13.463071 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:13.463054    2557 scope.go:117] "RemoveContainer" containerID="26da2311d92072bb7d41c8023a324af03eb40b1fda4a1917f77c3257b0175fb9"
Sep 14 13:14:13.463333 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:14:13.463314    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26da2311d92072bb7d41c8023a324af03eb40b1fda4a1917f77c3257b0175fb9\": container with ID starting with 26da2311d92072bb7d41c8023a324af03eb40b1fda4a1917f77c3257b0175fb9 not found: ID does not exist" containerID="26da2311d92072bb7d41c8023a324af03eb40b1fda4a1917f77c3257b0175fb9"
Sep 14 13:14:13.463404 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:13.463341    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26da2311d92072bb7d41c8023a324af03eb40b1fda4a1917f77c3257b0175fb9"} err="failed to get container status \"26da2311d92072bb7d41c8023a324af03eb40b1fda4a1917f77c3257b0175fb9\": rpc error: code = NotFound desc = could not find container \"26da2311d92072bb7d41c8023a324af03eb40b1fda4a1917f77c3257b0175fb9\": container with ID starting with 26da2311d92072bb7d41c8023a324af03eb40b1fda4a1917f77c3257b0175fb9 not found: ID does not exist"
Sep 14 13:14:13.463404 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:13.463358    2557 scope.go:117] "RemoveContainer" containerID="c2ae2e3ed70950c49bd782bba309053877cb4d6c2c65de5ed777fd3934a8a488"
Sep 14 13:14:13.473198 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:13.473174    2557 scope.go:117] "RemoveContainer" containerID="c2ae2e3ed70950c49bd782bba309053877cb4d6c2c65de5ed777fd3934a8a488"
Sep 14 13:14:13.473663 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:14:13.473644    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2ae2e3ed70950c49bd782bba309053877cb4d6c2c65de5ed777fd3934a8a488\": container with ID starting with c2ae2e3ed70950c49bd782bba309053877cb4d6c2c65de5ed777fd3934a8a488 not found: ID does not exist" containerID="c2ae2e3ed70950c49bd782bba309053877cb4d6c2c65de5ed777fd3934a8a488"
Sep 14 13:14:13.473759 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:13.473669    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ae2e3ed70950c49bd782bba309053877cb4d6c2c65de5ed777fd3934a8a488"} err="failed to get container status \"c2ae2e3ed70950c49bd782bba309053877cb4d6c2c65de5ed777fd3934a8a488\": rpc error: code = NotFound desc = could not find container \"c2ae2e3ed70950c49bd782bba309053877cb4d6c2c65de5ed777fd3934a8a488\": container with ID starting with c2ae2e3ed70950c49bd782bba309053877cb4d6c2c65de5ed777fd3934a8a488 not found: ID does not exist"
Sep 14 13:14:13.482209 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:13.482188    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt"]
Sep 14 13:14:13.487885 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:13.487863    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd15e28a-85f8649f55-tg5xt"]
Sep 14 13:14:13.501358 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:13.501340    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm"]
Sep 14 13:14:13.508160 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:13.508141    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd15e28a-6d4cdb9cf4-6m4dm"]
Sep 14 13:14:14.048847 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.048815    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="478bdff7-7449-47ea-8983-3be142ce4028" path="/var/lib/kubelet/pods/478bdff7-7449-47ea-8983-3be142ce4028/volumes"
Sep 14 13:14:14.049199 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.049184    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e58a84f-8110-4719-a251-47b1dff522e4" path="/var/lib/kubelet/pods/6e58a84f-8110-4719-a251-47b1dff522e4/volumes"
Sep 14 13:14:14.049473 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.049459    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c539994b-14df-493a-b009-32beb38a4e30" path="/var/lib/kubelet/pods/c539994b-14df-493a-b009-32beb38a4e30/volumes"
Sep 14 13:14:14.629786 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.629751    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"]
Sep 14 13:14:14.630369 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.630349    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="478bdff7-7449-47ea-8983-3be142ce4028" containerName="payload-pre-processing"
Sep 14 13:14:14.630437 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.630371    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="478bdff7-7449-47ea-8983-3be142ce4028" containerName="payload-pre-processing"
Sep 14 13:14:14.630437 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.630383    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e58a84f-8110-4719-a251-47b1dff522e4" containerName="payload-processing"
Sep 14 13:14:14.630437 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.630393    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e58a84f-8110-4719-a251-47b1dff522e4" containerName="payload-processing"
Sep 14 13:14:14.630437 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.630407    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c539994b-14df-493a-b009-32beb38a4e30" containerName="revoke-keys"
Sep 14 13:14:14.630437 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.630416    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c539994b-14df-493a-b009-32beb38a4e30" containerName="revoke-keys"
Sep 14 13:14:14.630693 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.630504    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e58a84f-8110-4719-a251-47b1dff522e4" containerName="payload-processing"
Sep 14 13:14:14.630693 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.630515    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c539994b-14df-493a-b009-32beb38a4e30" containerName="revoke-keys"
Sep 14 13:14:14.630693 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.630525    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="478bdff7-7449-47ea-8983-3be142ce4028" containerName="payload-pre-processing"
Sep 14 13:14:14.635657 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.635632    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.638915 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.638894    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-3448586b-openshift-default-dockercfg-pvdll\""
Sep 14 13:14:14.646754 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.646717    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"]
Sep 14 13:14:14.748902 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.748872    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istiod-ca-cert\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.749091 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.748907    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-podinfo\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.749091 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.748939    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-workload-socket\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.749091 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.748996    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-data\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.749091 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.749030    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-credential-socket\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.749358 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.749096    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pctcl\" (UniqueName: \"kubernetes.io/projected/e5690cfa-103a-4e3a-8655-8f6ef323cf11-kube-api-access-pctcl\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.749358 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.749152    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-workload-certs\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.749358 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.749212    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-token\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.749358 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.749240    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-envoy\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.849999 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.849951    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istiod-ca-cert\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.850172 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.850003    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-podinfo\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.850172 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.850053    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-workload-socket\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.850172 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.850092    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-data\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.850172 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.850132    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-credential-socket\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.850172 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.850160    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pctcl\" (UniqueName: \"kubernetes.io/projected/e5690cfa-103a-4e3a-8655-8f6ef323cf11-kube-api-access-pctcl\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.850434 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.850241    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-workload-certs\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.850434 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.850322    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-token\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.850857 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.850828    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-credential-socket\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.850967 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.850836    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istiod-ca-cert\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.850967 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.850353    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-envoy\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.851165 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.851146    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-workload-socket\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.851405 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.851373    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-data\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.851565 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.851528    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-workload-certs\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.853753 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.853714    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-envoy\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.854072 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.854049    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-podinfo\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.862463 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.862442    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pctcl\" (UniqueName: \"kubernetes.io/projected/e5690cfa-103a-4e3a-8655-8f6ef323cf11-kube-api-access-pctcl\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.862613 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.862579    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-token\") pod \"e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") " pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:14.948017 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:14.947995    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:15.233320 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:15.233298    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"]
Sep 14 13:14:15.234834 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:14:15.234805    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5690cfa_103a_4e3a_8655_8f6ef323cf11.slice/crio-32a2cee821a391c4d2ae7909073d5b233146e19268fa6451cb6f132275adacba WatchSource:0}: Error finding container 32a2cee821a391c4d2ae7909073d5b233146e19268fa6451cb6f132275adacba: Status 404 returned error can't find the container with id 32a2cee821a391c4d2ae7909073d5b233146e19268fa6451cb6f132275adacba
Sep 14 13:14:15.237430 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:15.237389    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:14:15.237504 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:15.237482    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:14:15.237545 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:15.237526    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:14:15.470859 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:15.470823    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr" event={"ID":"e5690cfa-103a-4e3a-8655-8f6ef323cf11","Type":"ContainerStarted","Data":"e79eff4f8b26e0f9b7d538e349aa841a0eadc0bfd5036ef91d8daf26ea074937"}
Sep 14 13:14:15.471030 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:15.470885    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr" event={"ID":"e5690cfa-103a-4e3a-8655-8f6ef323cf11","Type":"ContainerStarted","Data":"32a2cee821a391c4d2ae7909073d5b233146e19268fa6451cb6f132275adacba"}
Sep 14 13:14:15.494616 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:15.494514    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr" podStartSLOduration=1.494496486 podStartE2EDuration="1.494496486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:14:15.492241007 +0000 UTC m=+973.926916529" watchObservedRunningTime="2026-09-14 13:14:15.494496486 +0000 UTC m=+973.929172008"
Sep 14 13:14:15.948890 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:15.948864    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:16.953461 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:16.953371    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:17.481186 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:17.481156    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:17.481993 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:17.481973    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:18.987053 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:18.987019    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842"]
Sep 14 13:14:18.990679 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:18.990661    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842"
Sep 14 13:14:18.993654 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:18.993633    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-3448586b-dockercfg-lfltp\""
Sep 14 13:14:18.994035 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:18.994020    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3448586b\""
Sep 14 13:14:19.007433 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:19.007410    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842"]
Sep 14 13:14:19.089823 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:19.089799    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trljz\" (UniqueName: \"kubernetes.io/projected/d36ce920-f310-4614-9dc3-a9def28ed3f9-kube-api-access-trljz\") pod \"payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842\" (UID: \"d36ce920-f310-4614-9dc3-a9def28ed3f9\") " pod="openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842"
Sep 14 13:14:19.089989 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:19.089967    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d36ce920-f310-4614-9dc3-a9def28ed3f9-plugins-config-volume\") pod \"payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842\" (UID: \"d36ce920-f310-4614-9dc3-a9def28ed3f9\") " pod="openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842"
Sep 14 13:14:19.191180 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:19.191137    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trljz\" (UniqueName: \"kubernetes.io/projected/d36ce920-f310-4614-9dc3-a9def28ed3f9-kube-api-access-trljz\") pod \"payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842\" (UID: \"d36ce920-f310-4614-9dc3-a9def28ed3f9\") " pod="openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842"
Sep 14 13:14:19.191338 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:19.191236    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d36ce920-f310-4614-9dc3-a9def28ed3f9-plugins-config-volume\") pod \"payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842\" (UID: \"d36ce920-f310-4614-9dc3-a9def28ed3f9\") " pod="openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842"
Sep 14 13:14:19.191976 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:19.191957    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d36ce920-f310-4614-9dc3-a9def28ed3f9-plugins-config-volume\") pod \"payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842\" (UID: \"d36ce920-f310-4614-9dc3-a9def28ed3f9\") " pod="openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842"
Sep 14 13:14:19.200321 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:19.200298    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trljz\" (UniqueName: \"kubernetes.io/projected/d36ce920-f310-4614-9dc3-a9def28ed3f9-kube-api-access-trljz\") pod \"payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842\" (UID: \"d36ce920-f310-4614-9dc3-a9def28ed3f9\") " pod="openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842"
Sep 14 13:14:19.303910 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:19.303853    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842"
Sep 14 13:14:19.576319 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:19.576289    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842"]
Sep 14 13:14:19.578426 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:14:19.578397    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36ce920_f310_4614_9dc3_a9def28ed3f9.slice/crio-a780d44ce940c036a6bc06ae5850b67f9550a383a8abae09180d26a0fe0f7c8b WatchSource:0}: Error finding container a780d44ce940c036a6bc06ae5850b67f9550a383a8abae09180d26a0fe0f7c8b: Status 404 returned error can't find the container with id a780d44ce940c036a6bc06ae5850b67f9550a383a8abae09180d26a0fe0f7c8b
Sep 14 13:14:20.498610 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:20.498573    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842" event={"ID":"d36ce920-f310-4614-9dc3-a9def28ed3f9","Type":"ContainerStarted","Data":"bc3b225b8add791283b7fa6bcb266ae9ecc6fbc8a01462c49f123b5bf2d570cd"}
Sep 14 13:14:20.499100 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:20.498632    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842" event={"ID":"d36ce920-f310-4614-9dc3-a9def28ed3f9","Type":"ContainerStarted","Data":"a780d44ce940c036a6bc06ae5850b67f9550a383a8abae09180d26a0fe0f7c8b"}
Sep 14 13:14:20.499100 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:20.498686    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842"
Sep 14 13:14:20.518026 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:20.517970    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842" podStartSLOduration=2.517954649 podStartE2EDuration="2.517954649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:14:20.517512009 +0000 UTC m=+978.952187528" watchObservedRunningTime="2026-09-14 13:14:20.517954649 +0000 UTC m=+978.952630166"
Sep 14 13:14:21.696314 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:21.696275    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c14a5e-tx6bb"]
Sep 14 13:14:21.700468 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:21.700444    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c14a5e-tx6bb"
Sep 14 13:14:21.704024 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:21.703988    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbzp\""
Sep 14 13:14:21.705616 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:21.705590    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c14a5e-tx6bb"]
Sep 14 13:14:21.814549 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:21.814510    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cef7458-ff8a-4dc4-9860-e4544a867952-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-c14a5e-tx6bb\" (UID: \"1cef7458-ff8a-4dc4-9860-e4544a867952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c14a5e-tx6bb"
Sep 14 13:14:21.915132 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:21.915100    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cef7458-ff8a-4dc4-9860-e4544a867952-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-c14a5e-tx6bb\" (UID: \"1cef7458-ff8a-4dc4-9860-e4544a867952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c14a5e-tx6bb"
Sep 14 13:14:21.915706 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:21.915685    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cef7458-ff8a-4dc4-9860-e4544a867952-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-c14a5e-tx6bb\" (UID: \"1cef7458-ff8a-4dc4-9860-e4544a867952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c14a5e-tx6bb"
Sep 14 13:14:22.015286 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:22.015210    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c14a5e-tx6bb"
Sep 14 13:14:22.292647 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:22.292584    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c14a5e-tx6bb"]
Sep 14 13:14:22.294855 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:14:22.294829    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cef7458_ff8a_4dc4_9860_e4544a867952.slice/crio-2f98322a2ced422daf40bc9c316c0d4943ba667ad3e3ec1376fd0b1494bd2c74 WatchSource:0}: Error finding container 2f98322a2ced422daf40bc9c316c0d4943ba667ad3e3ec1376fd0b1494bd2c74: Status 404 returned error can't find the container with id 2f98322a2ced422daf40bc9c316c0d4943ba667ad3e3ec1376fd0b1494bd2c74
Sep 14 13:14:22.512367 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:22.512340    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c14a5e-tx6bb" event={"ID":"1cef7458-ff8a-4dc4-9860-e4544a867952","Type":"ContainerStarted","Data":"2f98322a2ced422daf40bc9c316c0d4943ba667ad3e3ec1376fd0b1494bd2c74"}
Sep 14 13:14:23.518661 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:23.518625    2557 generic.go:356] "Generic (PLEG): container finished" podID="1cef7458-ff8a-4dc4-9860-e4544a867952" containerID="346e396b0088f2d0e767cdc9263f5f3e8245916a61f19f7811130619b06c38f6" exitCode=0
Sep 14 13:14:23.519066 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:23.518673    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c14a5e-tx6bb" event={"ID":"1cef7458-ff8a-4dc4-9860-e4544a867952","Type":"ContainerDied","Data":"346e396b0088f2d0e767cdc9263f5f3e8245916a61f19f7811130619b06c38f6"}
Sep 14 13:14:24.680720 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:24.680686    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c14a5e-tx6bb"
Sep 14 13:14:24.843597 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:24.843515    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cef7458-ff8a-4dc4-9860-e4544a867952-maas-api-service-ca\") pod \"1cef7458-ff8a-4dc4-9860-e4544a867952\" (UID: \"1cef7458-ff8a-4dc4-9860-e4544a867952\") "
Sep 14 13:14:24.843961 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:24.843923    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cef7458-ff8a-4dc4-9860-e4544a867952-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1cef7458-ff8a-4dc4-9860-e4544a867952" (UID: "1cef7458-ff8a-4dc4-9860-e4544a867952"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:14:24.944962 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:24.944928    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cef7458-ff8a-4dc4-9860-e4544a867952-maas-api-service-ca\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:25.531026 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:25.531000    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c14a5e-tx6bb"
Sep 14 13:14:25.531192 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:25.531038    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c14a5e-tx6bb" event={"ID":"1cef7458-ff8a-4dc4-9860-e4544a867952","Type":"ContainerDied","Data":"2f98322a2ced422daf40bc9c316c0d4943ba667ad3e3ec1376fd0b1494bd2c74"}
Sep 14 13:14:25.531192 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:25.531083    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f98322a2ced422daf40bc9c316c0d4943ba667ad3e3ec1376fd0b1494bd2c74"
Sep 14 13:14:31.506915 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:31.506886    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842"
Sep 14 13:14:32.540883 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:32.540848    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv"]
Sep 14 13:14:32.541722 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:32.541683    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv" podUID="2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6" containerName="payload-processing" containerID="cri-o://f9f59cc2f09aafe2278ff40054ba053aaf5197537e5ef84a29091ee4af675f2c" gracePeriod=30
Sep 14 13:14:32.818770 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:32.818709    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c14a5e-tx6bb"]
Sep 14 13:14:32.822912 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:32.822883    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c14a5e-tx6bb"]
Sep 14 13:14:32.841624 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:32.841592    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv"
Sep 14 13:14:32.921358 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:32.921312    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6-plugins-config-volume\") pod \"2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6\" (UID: \"2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6\") "
Sep 14 13:14:32.921549 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:32.921436    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmnzn\" (UniqueName: \"kubernetes.io/projected/2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6-kube-api-access-rmnzn\") pod \"2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6\" (UID: \"2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6\") "
Sep 14 13:14:32.921717 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:32.921692    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6" (UID: "2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:14:32.921837 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:32.921821    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:32.923821 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:32.923789    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6-kube-api-access-rmnzn" (OuterVolumeSpecName: "kube-api-access-rmnzn") pod "2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6" (UID: "2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6"). InnerVolumeSpecName "kube-api-access-rmnzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:14:33.022391 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:33.022362    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmnzn\" (UniqueName: \"kubernetes.io/projected/2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6-kube-api-access-rmnzn\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:33.523884 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:33.523851    2557 patch_prober.go:28] interesting pod/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 14 13:14:33.524049 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:33.523907    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr" podUID="e5690cfa-103a-4e3a-8655-8f6ef323cf11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 14 13:14:33.606990 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:33.606961    2557 generic.go:356] "Generic (PLEG): container finished" podID="2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6" containerID="f9f59cc2f09aafe2278ff40054ba053aaf5197537e5ef84a29091ee4af675f2c" exitCode=0
Sep 14 13:14:33.607419 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:33.607011    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv"
Sep 14 13:14:33.607419 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:33.607041    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv" event={"ID":"2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6","Type":"ContainerDied","Data":"f9f59cc2f09aafe2278ff40054ba053aaf5197537e5ef84a29091ee4af675f2c"}
Sep 14 13:14:33.607419 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:33.607096    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv" event={"ID":"2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6","Type":"ContainerDied","Data":"2d705d6b25971004b7ebfce05a3115b9a01bb856011117fab58c73faac7487ce"}
Sep 14 13:14:33.607419 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:33.607120    2557 scope.go:117] "RemoveContainer" containerID="f9f59cc2f09aafe2278ff40054ba053aaf5197537e5ef84a29091ee4af675f2c"
Sep 14 13:14:33.616809 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:33.616791    2557 scope.go:117] "RemoveContainer" containerID="f9f59cc2f09aafe2278ff40054ba053aaf5197537e5ef84a29091ee4af675f2c"
Sep 14 13:14:33.617083 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:14:33.617067    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f59cc2f09aafe2278ff40054ba053aaf5197537e5ef84a29091ee4af675f2c\": container with ID starting with f9f59cc2f09aafe2278ff40054ba053aaf5197537e5ef84a29091ee4af675f2c not found: ID does not exist" containerID="f9f59cc2f09aafe2278ff40054ba053aaf5197537e5ef84a29091ee4af675f2c"
Sep 14 13:14:33.617138 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:33.617090    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f59cc2f09aafe2278ff40054ba053aaf5197537e5ef84a29091ee4af675f2c"} err="failed to get container status \"f9f59cc2f09aafe2278ff40054ba053aaf5197537e5ef84a29091ee4af675f2c\": rpc error: code = NotFound desc = could not find container \"f9f59cc2f09aafe2278ff40054ba053aaf5197537e5ef84a29091ee4af675f2c\": container with ID starting with f9f59cc2f09aafe2278ff40054ba053aaf5197537e5ef84a29091ee4af675f2c not found: ID does not exist"
Sep 14 13:14:33.629657 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:33.629633    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv"]
Sep 14 13:14:33.634767 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:33.632100    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c14a5e-5bddbc7695-f7hfv"]
Sep 14 13:14:34.051611 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:34.051574    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cef7458-ff8a-4dc4-9860-e4544a867952" path="/var/lib/kubelet/pods/1cef7458-ff8a-4dc4-9860-e4544a867952/volumes"
Sep 14 13:14:34.051971 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:34.051956    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6" path="/var/lib/kubelet/pods/2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6/volumes"
Sep 14 13:14:35.620345 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:35.620308    2557 generic.go:356] "Generic (PLEG): container finished" podID="e5690cfa-103a-4e3a-8655-8f6ef323cf11" containerID="e79eff4f8b26e0f9b7d538e349aa841a0eadc0bfd5036ef91d8daf26ea074937" exitCode=0
Sep 14 13:14:35.620809 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:35.620391    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr" event={"ID":"e5690cfa-103a-4e3a-8655-8f6ef323cf11","Type":"ContainerDied","Data":"e79eff4f8b26e0f9b7d538e349aa841a0eadc0bfd5036ef91d8daf26ea074937"}
Sep 14 13:14:35.620868 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:35.620817    2557 scope.go:117] "RemoveContainer" containerID="e79eff4f8b26e0f9b7d538e349aa841a0eadc0bfd5036ef91d8daf26ea074937"
Sep 14 13:14:35.622081 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:35.622049    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:14:35.622221 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:35.622130    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:14:35.622221 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:35.622200    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:14:36.561364 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:36.561318    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3448586b-d2zkb"]
Sep 14 13:14:36.562024 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:36.561996    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6" containerName="payload-processing"
Sep 14 13:14:36.562024 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:36.562024    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6" containerName="payload-processing"
Sep 14 13:14:36.562203 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:36.562052    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cef7458-ff8a-4dc4-9860-e4544a867952" containerName="revoke-keys"
Sep 14 13:14:36.562203 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:36.562061    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cef7458-ff8a-4dc4-9860-e4544a867952" containerName="revoke-keys"
Sep 14 13:14:36.562203 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:36.562159    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d9b0ef4-0f67-4a47-ad16-fc09fe0748d6" containerName="payload-processing"
Sep 14 13:14:36.562203 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:36.562173    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cef7458-ff8a-4dc4-9860-e4544a867952" containerName="revoke-keys"
Sep 14 13:14:36.565644 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:36.565616    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3448586b-d2zkb"
Sep 14 13:14:36.568589 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:36.568563    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbzp\""
Sep 14 13:14:36.579135 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:36.579109    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3448586b-d2zkb"]
Sep 14 13:14:36.627496 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:36.627469    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr" event={"ID":"e5690cfa-103a-4e3a-8655-8f6ef323cf11","Type":"ContainerStarted","Data":"2cc64e7a287e5c27174b40d765af3a84fae7e8a82a1c83ff3d24cade0996dcf2"}
Sep 14 13:14:36.658756 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:36.658710    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c4f650-56cb-48b5-bd36-72d7fa603da0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3448586b-d2zkb\" (UID: \"c3c4f650-56cb-48b5-bd36-72d7fa603da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3448586b-d2zkb"
Sep 14 13:14:36.761018 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:36.760975    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c4f650-56cb-48b5-bd36-72d7fa603da0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3448586b-d2zkb\" (UID: \"c3c4f650-56cb-48b5-bd36-72d7fa603da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3448586b-d2zkb"
Sep 14 13:14:36.761830 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:36.761802    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c4f650-56cb-48b5-bd36-72d7fa603da0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3448586b-d2zkb\" (UID: \"c3c4f650-56cb-48b5-bd36-72d7fa603da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3448586b-d2zkb"
Sep 14 13:14:36.882197 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:36.882119    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3448586b-d2zkb"
Sep 14 13:14:36.984490 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:36.952368    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:36.984490 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:36.952515    2557 patch_prober.go:28] interesting pod/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 14 13:14:36.984490 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:36.952572    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr" podUID="e5690cfa-103a-4e3a-8655-8f6ef323cf11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 14 13:14:37.159696 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:37.159097    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3448586b-d2zkb"]
Sep 14 13:14:37.634632 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:37.634581    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3448586b-d2zkb" event={"ID":"c3c4f650-56cb-48b5-bd36-72d7fa603da0","Type":"ContainerStarted","Data":"cee9c2cc8b1b0a9b9f9163e08c4bc524468d0b14c1ab33cb7eed889f086cf0eb"}
Sep 14 13:14:37.635060 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:37.634650    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3448586b-d2zkb" event={"ID":"c3c4f650-56cb-48b5-bd36-72d7fa603da0","Type":"ContainerStarted","Data":"a18641ff59df7957e51b0f8c8c6c61268ac1dae42569d562808406cb744f5c52"}
Sep 14 13:14:37.655057 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:37.653152    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3448586b-d2zkb" podStartSLOduration=1.65313235 podStartE2EDuration="1.65313235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:14:37.651527884 +0000 UTC m=+996.086203402" watchObservedRunningTime="2026-09-14 13:14:37.65313235 +0000 UTC m=+996.087807871"
Sep 14 13:14:37.948750 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:37.948696    2557 patch_prober.go:28] interesting pod/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 14 13:14:37.948919 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:37.948781    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr" podUID="e5690cfa-103a-4e3a-8655-8f6ef323cf11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 14 13:14:38.167570 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:38.167162    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"]
Sep 14 13:14:38.642000 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:38.641961    2557 generic.go:356] "Generic (PLEG): container finished" podID="c3c4f650-56cb-48b5-bd36-72d7fa603da0" containerID="cee9c2cc8b1b0a9b9f9163e08c4bc524468d0b14c1ab33cb7eed889f086cf0eb" exitCode=0
Sep 14 13:14:38.642358 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:38.642021    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3448586b-d2zkb" event={"ID":"c3c4f650-56cb-48b5-bd36-72d7fa603da0","Type":"ContainerDied","Data":"cee9c2cc8b1b0a9b9f9163e08c4bc524468d0b14c1ab33cb7eed889f086cf0eb"}
Sep 14 13:14:38.948663 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:38.948624    2557 patch_prober.go:28] interesting pod/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 14 13:14:38.948872 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:38.948718    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr" podUID="e5690cfa-103a-4e3a-8655-8f6ef323cf11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 14 13:14:39.821588 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:39.821567    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3448586b-d2zkb"
Sep 14 13:14:39.900140 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:39.900111    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c4f650-56cb-48b5-bd36-72d7fa603da0-maas-api-service-ca\") pod \"c3c4f650-56cb-48b5-bd36-72d7fa603da0\" (UID: \"c3c4f650-56cb-48b5-bd36-72d7fa603da0\") "
Sep 14 13:14:39.900682 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:39.900657    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c4f650-56cb-48b5-bd36-72d7fa603da0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c3c4f650-56cb-48b5-bd36-72d7fa603da0" (UID: "c3c4f650-56cb-48b5-bd36-72d7fa603da0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:14:39.954758 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:39.950096    2557 patch_prober.go:28] interesting pod/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 14 13:14:39.954758 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:39.950160    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr" podUID="e5690cfa-103a-4e3a-8655-8f6ef323cf11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 14 13:14:40.001596 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:40.001563    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c4f650-56cb-48b5-bd36-72d7fa603da0-maas-api-service-ca\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:40.660297 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:40.660204    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3448586b-d2zkb" event={"ID":"c3c4f650-56cb-48b5-bd36-72d7fa603da0","Type":"ContainerDied","Data":"a18641ff59df7957e51b0f8c8c6c61268ac1dae42569d562808406cb744f5c52"}
Sep 14 13:14:40.660297 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:40.660272    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a18641ff59df7957e51b0f8c8c6c61268ac1dae42569d562808406cb744f5c52"
Sep 14 13:14:40.660605 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:40.660341    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3448586b-d2zkb"
Sep 14 13:14:40.948509 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:40.948470    2557 patch_prober.go:28] interesting pod/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 14 13:14:40.949199 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:40.948540    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr" podUID="e5690cfa-103a-4e3a-8655-8f6ef323cf11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 14 13:14:41.953649 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:41.953618    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:41.954086 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:41.954065    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:41.954847 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:41.954825    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:47.313858 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:47.313825    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842"]
Sep 14 13:14:47.314328 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:47.314075    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842" podUID="d36ce920-f310-4614-9dc3-a9def28ed3f9" containerName="payload-processing" containerID="cri-o://bc3b225b8add791283b7fa6bcb266ae9ecc6fbc8a01462c49f123b5bf2d570cd" gracePeriod=30
Sep 14 13:14:47.519430 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:47.519399    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3448586b-d2zkb"]
Sep 14 13:14:47.521791 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:47.521762    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3448586b-d2zkb"]
Sep 14 13:14:47.580133 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:47.580108    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842"
Sep 14 13:14:47.672313 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:47.672278    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trljz\" (UniqueName: \"kubernetes.io/projected/d36ce920-f310-4614-9dc3-a9def28ed3f9-kube-api-access-trljz\") pod \"d36ce920-f310-4614-9dc3-a9def28ed3f9\" (UID: \"d36ce920-f310-4614-9dc3-a9def28ed3f9\") "
Sep 14 13:14:47.672497 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:47.672385    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d36ce920-f310-4614-9dc3-a9def28ed3f9-plugins-config-volume\") pod \"d36ce920-f310-4614-9dc3-a9def28ed3f9\" (UID: \"d36ce920-f310-4614-9dc3-a9def28ed3f9\") "
Sep 14 13:14:47.672892 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:47.672859    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d36ce920-f310-4614-9dc3-a9def28ed3f9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d36ce920-f310-4614-9dc3-a9def28ed3f9" (UID: "d36ce920-f310-4614-9dc3-a9def28ed3f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:14:47.675218 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:47.675190    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36ce920-f310-4614-9dc3-a9def28ed3f9-kube-api-access-trljz" (OuterVolumeSpecName: "kube-api-access-trljz") pod "d36ce920-f310-4614-9dc3-a9def28ed3f9" (UID: "d36ce920-f310-4614-9dc3-a9def28ed3f9"). InnerVolumeSpecName "kube-api-access-trljz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:14:47.685466 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:47.679726    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"]
Sep 14 13:14:47.685466 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:47.681234    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr" podUID="e5690cfa-103a-4e3a-8655-8f6ef323cf11" containerName="istio-proxy" containerID="cri-o://2cc64e7a287e5c27174b40d765af3a84fae7e8a82a1c83ff3d24cade0996dcf2" gracePeriod=30
Sep 14 13:14:47.700824 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:47.700799    2557 generic.go:356] "Generic (PLEG): container finished" podID="d36ce920-f310-4614-9dc3-a9def28ed3f9" containerID="bc3b225b8add791283b7fa6bcb266ae9ecc6fbc8a01462c49f123b5bf2d570cd" exitCode=0
Sep 14 13:14:47.700938 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:47.700837    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842" event={"ID":"d36ce920-f310-4614-9dc3-a9def28ed3f9","Type":"ContainerDied","Data":"bc3b225b8add791283b7fa6bcb266ae9ecc6fbc8a01462c49f123b5bf2d570cd"}
Sep 14 13:14:47.700938 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:47.700848    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842"
Sep 14 13:14:47.700938 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:47.700874    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842" event={"ID":"d36ce920-f310-4614-9dc3-a9def28ed3f9","Type":"ContainerDied","Data":"a780d44ce940c036a6bc06ae5850b67f9550a383a8abae09180d26a0fe0f7c8b"}
Sep 14 13:14:47.700938 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:47.700896    2557 scope.go:117] "RemoveContainer" containerID="bc3b225b8add791283b7fa6bcb266ae9ecc6fbc8a01462c49f123b5bf2d570cd"
Sep 14 13:14:47.710230 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:47.710208    2557 scope.go:117] "RemoveContainer" containerID="bc3b225b8add791283b7fa6bcb266ae9ecc6fbc8a01462c49f123b5bf2d570cd"
Sep 14 13:14:47.710526 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:14:47.710500    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc3b225b8add791283b7fa6bcb266ae9ecc6fbc8a01462c49f123b5bf2d570cd\": container with ID starting with bc3b225b8add791283b7fa6bcb266ae9ecc6fbc8a01462c49f123b5bf2d570cd not found: ID does not exist" containerID="bc3b225b8add791283b7fa6bcb266ae9ecc6fbc8a01462c49f123b5bf2d570cd"
Sep 14 13:14:47.710573 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:47.710532    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3b225b8add791283b7fa6bcb266ae9ecc6fbc8a01462c49f123b5bf2d570cd"} err="failed to get container status \"bc3b225b8add791283b7fa6bcb266ae9ecc6fbc8a01462c49f123b5bf2d570cd\": rpc error: code = NotFound desc = could not find container \"bc3b225b8add791283b7fa6bcb266ae9ecc6fbc8a01462c49f123b5bf2d570cd\": container with ID starting with bc3b225b8add791283b7fa6bcb266ae9ecc6fbc8a01462c49f123b5bf2d570cd not found: ID does not exist"
Sep 14 13:14:47.728411 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:47.728385    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842"]
Sep 14 13:14:47.730246 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:47.730225    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3448586b-74b69d6dfc-l2842"]
Sep 14 13:14:47.781465 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:47.781425    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trljz\" (UniqueName: \"kubernetes.io/projected/d36ce920-f310-4614-9dc3-a9def28ed3f9-kube-api-access-trljz\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:47.781662 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:47.781461    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d36ce920-f310-4614-9dc3-a9def28ed3f9-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:48.049309 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:48.049273    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3c4f650-56cb-48b5-bd36-72d7fa603da0" path="/var/lib/kubelet/pods/c3c4f650-56cb-48b5-bd36-72d7fa603da0/volumes"
Sep 14 13:14:48.049644 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:48.049629    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d36ce920-f310-4614-9dc3-a9def28ed3f9" path="/var/lib/kubelet/pods/d36ce920-f310-4614-9dc3-a9def28ed3f9/volumes"
Sep 14 13:14:52.936387 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:52.936363    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:53.031668 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.031603    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-workload-socket\") pod \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") "
Sep 14 13:14:53.031668 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.031638    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-token\") pod \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") "
Sep 14 13:14:53.031668 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.031665    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-credential-socket\") pod \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") "
Sep 14 13:14:53.031914 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.031711    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-podinfo\") pod \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") "
Sep 14 13:14:53.031914 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.031763    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-data\") pod \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") "
Sep 14 13:14:53.031914 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.031895    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istiod-ca-cert\") pod \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") "
Sep 14 13:14:53.032042 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.031908    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e5690cfa-103a-4e3a-8655-8f6ef323cf11" (UID: "e5690cfa-103a-4e3a-8655-8f6ef323cf11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:14:53.032042 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.032006    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pctcl\" (UniqueName: \"kubernetes.io/projected/e5690cfa-103a-4e3a-8655-8f6ef323cf11-kube-api-access-pctcl\") pod \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") "
Sep 14 13:14:53.032042 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.032030    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e5690cfa-103a-4e3a-8655-8f6ef323cf11" (UID: "e5690cfa-103a-4e3a-8655-8f6ef323cf11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:14:53.032182 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.032047    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-data" (OuterVolumeSpecName: "istio-data") pod "e5690cfa-103a-4e3a-8655-8f6ef323cf11" (UID: "e5690cfa-103a-4e3a-8655-8f6ef323cf11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:14:53.032182 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.032054    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-workload-certs\") pod \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") "
Sep 14 13:14:53.032182 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.032160    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-envoy\") pod \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\" (UID: \"e5690cfa-103a-4e3a-8655-8f6ef323cf11\") "
Sep 14 13:14:53.032330 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.032210    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e5690cfa-103a-4e3a-8655-8f6ef323cf11" (UID: "e5690cfa-103a-4e3a-8655-8f6ef323cf11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:14:53.032330 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.032250    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e5690cfa-103a-4e3a-8655-8f6ef323cf11" (UID: "e5690cfa-103a-4e3a-8655-8f6ef323cf11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:14:53.032523 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.032503    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-credential-socket\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:53.032584 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.032528    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-data\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:53.032584 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.032543    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istiod-ca-cert\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:53.032584 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.032555    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-workload-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:53.032584 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.032568    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-workload-socket\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:53.033987 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.033963    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-token" (OuterVolumeSpecName: "istio-token") pod "e5690cfa-103a-4e3a-8655-8f6ef323cf11" (UID: "e5690cfa-103a-4e3a-8655-8f6ef323cf11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:14:53.033987 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.033975    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5690cfa-103a-4e3a-8655-8f6ef323cf11-kube-api-access-pctcl" (OuterVolumeSpecName: "kube-api-access-pctcl") pod "e5690cfa-103a-4e3a-8655-8f6ef323cf11" (UID: "e5690cfa-103a-4e3a-8655-8f6ef323cf11"). InnerVolumeSpecName "kube-api-access-pctcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:14:53.034140 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.034034    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e5690cfa-103a-4e3a-8655-8f6ef323cf11" (UID: "e5690cfa-103a-4e3a-8655-8f6ef323cf11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 13:14:53.034140 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.034130    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e5690cfa-103a-4e3a-8655-8f6ef323cf11" (UID: "e5690cfa-103a-4e3a-8655-8f6ef323cf11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:14:53.133967 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.133934    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-token\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:53.133967 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.133965    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-podinfo\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:53.134184 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.133978    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pctcl\" (UniqueName: \"kubernetes.io/projected/e5690cfa-103a-4e3a-8655-8f6ef323cf11-kube-api-access-pctcl\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:53.134184 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.133992    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5690cfa-103a-4e3a-8655-8f6ef323cf11-istio-envoy\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:53.189918 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.189871    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd" podUID="bad29697-772e-458b-a7cf-38e441ccf37f" containerName="main" containerID="cri-o://af2c5145cdf05d85d1a8b283795b0f76d2c8f4f2a91d93a19a5fb3902b02f566" gracePeriod=45
Sep 14 13:14:53.424366 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.424343    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:14:53.537343 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.537317    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-home\") pod \"bad29697-772e-458b-a7cf-38e441ccf37f\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") "
Sep 14 13:14:53.537498 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.537358    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-254hf\" (UniqueName: \"kubernetes.io/projected/bad29697-772e-458b-a7cf-38e441ccf37f-kube-api-access-254hf\") pod \"bad29697-772e-458b-a7cf-38e441ccf37f\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") "
Sep 14 13:14:53.537498 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.537405    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-tmp-dir\") pod \"bad29697-772e-458b-a7cf-38e441ccf37f\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") "
Sep 14 13:14:53.537498 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.537427    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-model-cache\") pod \"bad29697-772e-458b-a7cf-38e441ccf37f\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") "
Sep 14 13:14:53.537671 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.537500    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-dshm\") pod \"bad29697-772e-458b-a7cf-38e441ccf37f\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") "
Sep 14 13:14:53.537671 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.537530    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bad29697-772e-458b-a7cf-38e441ccf37f-tls-certs\") pod \"bad29697-772e-458b-a7cf-38e441ccf37f\" (UID: \"bad29697-772e-458b-a7cf-38e441ccf37f\") "
Sep 14 13:14:53.537671 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.537601    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-home" (OuterVolumeSpecName: "home") pod "bad29697-772e-458b-a7cf-38e441ccf37f" (UID: "bad29697-772e-458b-a7cf-38e441ccf37f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:14:53.537671 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.537616    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bad29697-772e-458b-a7cf-38e441ccf37f" (UID: "bad29697-772e-458b-a7cf-38e441ccf37f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:14:53.537908 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.537764    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-model-cache" (OuterVolumeSpecName: "model-cache") pod "bad29697-772e-458b-a7cf-38e441ccf37f" (UID: "bad29697-772e-458b-a7cf-38e441ccf37f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:14:53.537964 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.537911    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-tmp-dir\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:53.537964 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.537926    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-model-cache\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:53.537964 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.537934    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-home\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:53.539628 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.539602    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bad29697-772e-458b-a7cf-38e441ccf37f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bad29697-772e-458b-a7cf-38e441ccf37f" (UID: "bad29697-772e-458b-a7cf-38e441ccf37f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:14:53.539734 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.539666    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-dshm" (OuterVolumeSpecName: "dshm") pod "bad29697-772e-458b-a7cf-38e441ccf37f" (UID: "bad29697-772e-458b-a7cf-38e441ccf37f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:14:53.539734 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.539691    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad29697-772e-458b-a7cf-38e441ccf37f-kube-api-access-254hf" (OuterVolumeSpecName: "kube-api-access-254hf") pod "bad29697-772e-458b-a7cf-38e441ccf37f" (UID: "bad29697-772e-458b-a7cf-38e441ccf37f"). InnerVolumeSpecName "kube-api-access-254hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:14:53.638680 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.638622    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bad29697-772e-458b-a7cf-38e441ccf37f-dshm\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:53.638680 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.638643    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bad29697-772e-458b-a7cf-38e441ccf37f-tls-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:53.638680 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.638653    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-254hf\" (UniqueName: \"kubernetes.io/projected/bad29697-772e-458b-a7cf-38e441ccf37f-kube-api-access-254hf\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:14:53.734808 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.734779    2557 generic.go:356] "Generic (PLEG): container finished" podID="e5690cfa-103a-4e3a-8655-8f6ef323cf11" containerID="2cc64e7a287e5c27174b40d765af3a84fae7e8a82a1c83ff3d24cade0996dcf2" exitCode=0
Sep 14 13:14:53.734952 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.734840    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"
Sep 14 13:14:53.734952 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.734862    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr" event={"ID":"e5690cfa-103a-4e3a-8655-8f6ef323cf11","Type":"ContainerDied","Data":"2cc64e7a287e5c27174b40d765af3a84fae7e8a82a1c83ff3d24cade0996dcf2"}
Sep 14 13:14:53.735070 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.734950    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr" event={"ID":"e5690cfa-103a-4e3a-8655-8f6ef323cf11","Type":"ContainerDied","Data":"32a2cee821a391c4d2ae7909073d5b233146e19268fa6451cb6f132275adacba"}
Sep 14 13:14:53.735070 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.734982    2557 scope.go:117] "RemoveContainer" containerID="2cc64e7a287e5c27174b40d765af3a84fae7e8a82a1c83ff3d24cade0996dcf2"
Sep 14 13:14:53.736298 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.736280    2557 generic.go:356] "Generic (PLEG): container finished" podID="bad29697-772e-458b-a7cf-38e441ccf37f" containerID="af2c5145cdf05d85d1a8b283795b0f76d2c8f4f2a91d93a19a5fb3902b02f566" exitCode=0
Sep 14 13:14:53.736387 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.736323    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"
Sep 14 13:14:53.736440 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.736317    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd" event={"ID":"bad29697-772e-458b-a7cf-38e441ccf37f","Type":"ContainerDied","Data":"af2c5145cdf05d85d1a8b283795b0f76d2c8f4f2a91d93a19a5fb3902b02f566"}
Sep 14 13:14:53.736490 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.736436    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd" event={"ID":"bad29697-772e-458b-a7cf-38e441ccf37f","Type":"ContainerDied","Data":"5e6d2a686d9f81630f3d5b15b44921e23ce33a4ac0767e606fe55cf9e0822289"}
Sep 14 13:14:53.745601 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.745572    2557 scope.go:117] "RemoveContainer" containerID="e79eff4f8b26e0f9b7d538e349aa841a0eadc0bfd5036ef91d8daf26ea074937"
Sep 14 13:14:53.753910 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.753832    2557 scope.go:117] "RemoveContainer" containerID="2cc64e7a287e5c27174b40d765af3a84fae7e8a82a1c83ff3d24cade0996dcf2"
Sep 14 13:14:53.754110 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:14:53.754092    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cc64e7a287e5c27174b40d765af3a84fae7e8a82a1c83ff3d24cade0996dcf2\": container with ID starting with 2cc64e7a287e5c27174b40d765af3a84fae7e8a82a1c83ff3d24cade0996dcf2 not found: ID does not exist" containerID="2cc64e7a287e5c27174b40d765af3a84fae7e8a82a1c83ff3d24cade0996dcf2"
Sep 14 13:14:53.754169 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.754117    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cc64e7a287e5c27174b40d765af3a84fae7e8a82a1c83ff3d24cade0996dcf2"} err="failed to get container status \"2cc64e7a287e5c27174b40d765af3a84fae7e8a82a1c83ff3d24cade0996dcf2\": rpc error: code = NotFound desc = could not find container \"2cc64e7a287e5c27174b40d765af3a84fae7e8a82a1c83ff3d24cade0996dcf2\": container with ID starting with 2cc64e7a287e5c27174b40d765af3a84fae7e8a82a1c83ff3d24cade0996dcf2 not found: ID does not exist"
Sep 14 13:14:53.754169 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.754133    2557 scope.go:117] "RemoveContainer" containerID="e79eff4f8b26e0f9b7d538e349aa841a0eadc0bfd5036ef91d8daf26ea074937"
Sep 14 13:14:53.754368 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:14:53.754350    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e79eff4f8b26e0f9b7d538e349aa841a0eadc0bfd5036ef91d8daf26ea074937\": container with ID starting with e79eff4f8b26e0f9b7d538e349aa841a0eadc0bfd5036ef91d8daf26ea074937 not found: ID does not exist" containerID="e79eff4f8b26e0f9b7d538e349aa841a0eadc0bfd5036ef91d8daf26ea074937"
Sep 14 13:14:53.754428 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.754374    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e79eff4f8b26e0f9b7d538e349aa841a0eadc0bfd5036ef91d8daf26ea074937"} err="failed to get container status \"e79eff4f8b26e0f9b7d538e349aa841a0eadc0bfd5036ef91d8daf26ea074937\": rpc error: code = NotFound desc = could not find container \"e79eff4f8b26e0f9b7d538e349aa841a0eadc0bfd5036ef91d8daf26ea074937\": container with ID starting with e79eff4f8b26e0f9b7d538e349aa841a0eadc0bfd5036ef91d8daf26ea074937 not found: ID does not exist"
Sep 14 13:14:53.754428 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.754390    2557 scope.go:117] "RemoveContainer" containerID="af2c5145cdf05d85d1a8b283795b0f76d2c8f4f2a91d93a19a5fb3902b02f566"
Sep 14 13:14:53.759294 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.759275    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"]
Sep 14 13:14:53.761158 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.761137    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-3448586b-openshift-default-68cf5c59bd-gkvhr"]
Sep 14 13:14:53.763153 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.763139    2557 scope.go:117] "RemoveContainer" containerID="af2c5145cdf05d85d1a8b283795b0f76d2c8f4f2a91d93a19a5fb3902b02f566"
Sep 14 13:14:53.763401 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:14:53.763384    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af2c5145cdf05d85d1a8b283795b0f76d2c8f4f2a91d93a19a5fb3902b02f566\": container with ID starting with af2c5145cdf05d85d1a8b283795b0f76d2c8f4f2a91d93a19a5fb3902b02f566 not found: ID does not exist" containerID="af2c5145cdf05d85d1a8b283795b0f76d2c8f4f2a91d93a19a5fb3902b02f566"
Sep 14 13:14:53.763452 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.763406    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af2c5145cdf05d85d1a8b283795b0f76d2c8f4f2a91d93a19a5fb3902b02f566"} err="failed to get container status \"af2c5145cdf05d85d1a8b283795b0f76d2c8f4f2a91d93a19a5fb3902b02f566\": rpc error: code = NotFound desc = could not find container \"af2c5145cdf05d85d1a8b283795b0f76d2c8f4f2a91d93a19a5fb3902b02f566\": container with ID starting with af2c5145cdf05d85d1a8b283795b0f76d2c8f4f2a91d93a19a5fb3902b02f566 not found: ID does not exist"
Sep 14 13:14:53.771447 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.771418    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"]
Sep 14 13:14:53.775417 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:53.775396    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-c14a5e/autores-model-c14a5e-kserve-58dfddb4bc-btfkd"]
Sep 14 13:14:54.048970 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:54.048941    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bad29697-772e-458b-a7cf-38e441ccf37f" path="/var/lib/kubelet/pods/bad29697-772e-458b-a7cf-38e441ccf37f/volumes"
Sep 14 13:14:54.049353 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:54.049339    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5690cfa-103a-4e3a-8655-8f6ef323cf11" path="/var/lib/kubelet/pods/e5690cfa-103a-4e3a-8655-8f6ef323cf11/volumes"
Sep 14 13:14:57.032977 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.032950    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"]
Sep 14 13:14:57.033489 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.033471    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bad29697-772e-458b-a7cf-38e441ccf37f" containerName="main"
Sep 14 13:14:57.033586 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.033494    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bad29697-772e-458b-a7cf-38e441ccf37f" containerName="main"
Sep 14 13:14:57.033586 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.033514    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3c4f650-56cb-48b5-bd36-72d7fa603da0" containerName="revoke-keys"
Sep 14 13:14:57.033586 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.033523    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3c4f650-56cb-48b5-bd36-72d7fa603da0" containerName="revoke-keys"
Sep 14 13:14:57.033586 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.033532    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d36ce920-f310-4614-9dc3-a9def28ed3f9" containerName="payload-processing"
Sep 14 13:14:57.033586 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.033541    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d36ce920-f310-4614-9dc3-a9def28ed3f9" containerName="payload-processing"
Sep 14 13:14:57.033586 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.033564    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5690cfa-103a-4e3a-8655-8f6ef323cf11" containerName="istio-proxy"
Sep 14 13:14:57.033586 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.033572    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5690cfa-103a-4e3a-8655-8f6ef323cf11" containerName="istio-proxy"
Sep 14 13:14:57.033586 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.033585    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5690cfa-103a-4e3a-8655-8f6ef323cf11" containerName="istio-proxy"
Sep 14 13:14:57.034010 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.033593    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5690cfa-103a-4e3a-8655-8f6ef323cf11" containerName="istio-proxy"
Sep 14 13:14:57.034010 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.033692    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5690cfa-103a-4e3a-8655-8f6ef323cf11" containerName="istio-proxy"
Sep 14 13:14:57.034010 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.033703    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d36ce920-f310-4614-9dc3-a9def28ed3f9" containerName="payload-processing"
Sep 14 13:14:57.034010 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.033713    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5690cfa-103a-4e3a-8655-8f6ef323cf11" containerName="istio-proxy"
Sep 14 13:14:57.034010 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.033726    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bad29697-772e-458b-a7cf-38e441ccf37f" containerName="main"
Sep 14 13:14:57.034010 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.033734    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3c4f650-56cb-48b5-bd36-72d7fa603da0" containerName="revoke-keys"
Sep 14 13:14:57.038383 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.038365    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:14:57.042990 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.042965    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-bc851d\"/\"openshift-service-ca.crt\""
Sep 14 13:14:57.043102 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.043088    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-bc851d\"/\"kube-root-ca.crt\""
Sep 14 13:14:57.043188 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.043090    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-bc851d\"/\"default-dockercfg-xtjzp\""
Sep 14 13:14:57.043249 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.043220    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-bc851d\"/\"premium-w0-bc851d-kserve-self-signed-certs\""
Sep 14 13:14:57.052472 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.052451    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"]
Sep 14 13:14:57.167563 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.167533    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-tls-certs\") pod \"premium-w0-bc851d-kserve-6d457757c9-2r9sl\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") " pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:14:57.167563 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.167566    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44f89\" (UniqueName: \"kubernetes.io/projected/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-kube-api-access-44f89\") pod \"premium-w0-bc851d-kserve-6d457757c9-2r9sl\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") " pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:14:57.167804 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.167622    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-tmp-dir\") pod \"premium-w0-bc851d-kserve-6d457757c9-2r9sl\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") " pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:14:57.167804 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.167660    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-home\") pod \"premium-w0-bc851d-kserve-6d457757c9-2r9sl\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") " pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:14:57.167804 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.167721    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-model-cache\") pod \"premium-w0-bc851d-kserve-6d457757c9-2r9sl\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") " pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:14:57.167933 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.167814    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-dshm\") pod \"premium-w0-bc851d-kserve-6d457757c9-2r9sl\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") " pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:14:57.268655 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.268621    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-dshm\") pod \"premium-w0-bc851d-kserve-6d457757c9-2r9sl\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") " pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:14:57.268874 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.268676    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-tls-certs\") pod \"premium-w0-bc851d-kserve-6d457757c9-2r9sl\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") " pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:14:57.268874 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.268801    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44f89\" (UniqueName: \"kubernetes.io/projected/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-kube-api-access-44f89\") pod \"premium-w0-bc851d-kserve-6d457757c9-2r9sl\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") " pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:14:57.269016 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.268884    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-tmp-dir\") pod \"premium-w0-bc851d-kserve-6d457757c9-2r9sl\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") " pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:14:57.269016 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.268932    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-home\") pod \"premium-w0-bc851d-kserve-6d457757c9-2r9sl\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") " pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:14:57.269016 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.268985    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-model-cache\") pod \"premium-w0-bc851d-kserve-6d457757c9-2r9sl\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") " pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:14:57.269271 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.269242    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-tmp-dir\") pod \"premium-w0-bc851d-kserve-6d457757c9-2r9sl\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") " pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:14:57.269381 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.269278    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-home\") pod \"premium-w0-bc851d-kserve-6d457757c9-2r9sl\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") " pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:14:57.269381 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.269293    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-model-cache\") pod \"premium-w0-bc851d-kserve-6d457757c9-2r9sl\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") " pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:14:57.271039 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.271012    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-dshm\") pod \"premium-w0-bc851d-kserve-6d457757c9-2r9sl\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") " pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:14:57.271320 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.271301    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-tls-certs\") pod \"premium-w0-bc851d-kserve-6d457757c9-2r9sl\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") " pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:14:57.279604 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.279572    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44f89\" (UniqueName: \"kubernetes.io/projected/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-kube-api-access-44f89\") pod \"premium-w0-bc851d-kserve-6d457757c9-2r9sl\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") " pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:14:57.347706 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.347641    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:14:57.619519 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.619493    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"]
Sep 14 13:14:57.622427 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:14:57.622399    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12b7d6dd_2596_4576_8fd1_bcd213ecf6e5.slice/crio-3506e93a03265220f5c340fd41f5ef3cae0f4732b25a03b48f0e1d1cf3b5a5d9 WatchSource:0}: Error finding container 3506e93a03265220f5c340fd41f5ef3cae0f4732b25a03b48f0e1d1cf3b5a5d9: Status 404 returned error can't find the container with id 3506e93a03265220f5c340fd41f5ef3cae0f4732b25a03b48f0e1d1cf3b5a5d9
Sep 14 13:14:57.764646 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.764611    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl" event={"ID":"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5","Type":"ContainerStarted","Data":"a87e76043568306fc0d6734335987378fefaab3db743d13b77ed107c894d5dcb"}
Sep 14 13:14:57.764822 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.764660    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl" event={"ID":"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5","Type":"ContainerStarted","Data":"3506e93a03265220f5c340fd41f5ef3cae0f4732b25a03b48f0e1d1cf3b5a5d9"}
Sep 14 13:14:57.809996 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:14:57.809949    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl" podStartSLOduration=0.8099339 podStartE2EDuration="809.9339ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:14:57.808637208 +0000 UTC m=+1016.243312724" watchObservedRunningTime="2026-09-14 13:14:57.8099339 +0000 UTC m=+1016.244609419"
Sep 14 13:15:00.109753 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.109703    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"]
Sep 14 13:15:00.113778 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.113755    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:00.121784 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.121734    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b3d6c5\"/\"kube-root-ca.crt\""
Sep 14 13:15:00.122015 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.121996    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b3d6c5\"/\"premium-w4-b3d6c5-kserve-self-signed-certs\""
Sep 14 13:15:00.122154 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.122013    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b3d6c5\"/\"default-dockercfg-lpgkm\""
Sep 14 13:15:00.122154 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.122095    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b3d6c5\"/\"openshift-service-ca.crt\""
Sep 14 13:15:00.134028 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.134006    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"]
Sep 14 13:15:00.194896 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.194872    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-model-cache\") pod \"premium-w4-b3d6c5-kserve-6cb94675b7-8j44q\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") " pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:00.195050 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.194962    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-home\") pod \"premium-w4-b3d6c5-kserve-6cb94675b7-8j44q\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") " pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:00.195050 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.195033    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7kfg\" (UniqueName: \"kubernetes.io/projected/7634a83e-5864-499a-9769-f29de26d5ad5-kube-api-access-c7kfg\") pod \"premium-w4-b3d6c5-kserve-6cb94675b7-8j44q\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") " pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:00.195173 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.195071    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7634a83e-5864-499a-9769-f29de26d5ad5-tls-certs\") pod \"premium-w4-b3d6c5-kserve-6cb94675b7-8j44q\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") " pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:00.195173 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.195125    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-dshm\") pod \"premium-w4-b3d6c5-kserve-6cb94675b7-8j44q\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") " pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:00.195173 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.195159    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-tmp-dir\") pod \"premium-w4-b3d6c5-kserve-6cb94675b7-8j44q\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") " pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:00.204075 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.204046    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cadd9c-29823195-2w9qz"]
Sep 14 13:15:00.207510 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.207494    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc851d-29823195-t4bjm"]
Sep 14 13:15:00.207621 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.207605    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cadd9c-29823195-2w9qz"
Sep 14 13:15:00.210514 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.210497    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbzp\""
Sep 14 13:15:00.210928 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.210910    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-975227-29823195-rk8r7"]
Sep 14 13:15:00.211030 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.211020    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc851d-29823195-t4bjm"
Sep 14 13:15:00.214055 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.214036    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-975227-29823195-rk8r7"
Sep 14 13:15:00.219201 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.219180    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cadd9c-29823195-2w9qz"]
Sep 14 13:15:00.220325 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.220185    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cadd9c-29823195-2w9qz"
Sep 14 13:15:00.223268 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.222769    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc851d-29823195-t4bjm"]
Sep 14 13:15:00.227826 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.225241    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-975227-29823195-rk8r7"]
Sep 14 13:15:00.233691 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.233673    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc851d-29823195-t4bjm"
Sep 14 13:15:00.274325 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.274279    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-975227-29823195-rk8r7"
Sep 14 13:15:00.297064 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.297016    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7634a83e-5864-499a-9769-f29de26d5ad5-tls-certs\") pod \"premium-w4-b3d6c5-kserve-6cb94675b7-8j44q\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") " pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:00.298877 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.297077    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-dshm\") pod \"premium-w4-b3d6c5-kserve-6cb94675b7-8j44q\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") " pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:00.298877 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.298650    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-tmp-dir\") pod \"premium-w4-b3d6c5-kserve-6cb94675b7-8j44q\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") " pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:00.298877 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.298699    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-model-cache\") pod \"premium-w4-b3d6c5-kserve-6cb94675b7-8j44q\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") " pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:00.303310 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.299168    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-tmp-dir\") pod \"premium-w4-b3d6c5-kserve-6cb94675b7-8j44q\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") " pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:00.303310 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.299344    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-home\") pod \"premium-w4-b3d6c5-kserve-6cb94675b7-8j44q\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") " pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:00.303310 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.299393    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7kfg\" (UniqueName: \"kubernetes.io/projected/7634a83e-5864-499a-9769-f29de26d5ad5-kube-api-access-c7kfg\") pod \"premium-w4-b3d6c5-kserve-6cb94675b7-8j44q\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") " pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:00.303310 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.299966    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-model-cache\") pod \"premium-w4-b3d6c5-kserve-6cb94675b7-8j44q\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") " pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:00.303310 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.301163    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7634a83e-5864-499a-9769-f29de26d5ad5-tls-certs\") pod \"premium-w4-b3d6c5-kserve-6cb94675b7-8j44q\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") " pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:00.303310 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.302186    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-home\") pod \"premium-w4-b3d6c5-kserve-6cb94675b7-8j44q\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") " pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:00.303310 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.302915    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-dshm\") pod \"premium-w4-b3d6c5-kserve-6cb94675b7-8j44q\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") " pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:00.312656 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.312598    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7kfg\" (UniqueName: \"kubernetes.io/projected/7634a83e-5864-499a-9769-f29de26d5ad5-kube-api-access-c7kfg\") pod \"premium-w4-b3d6c5-kserve-6cb94675b7-8j44q\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") " pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:00.424869 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.424842    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:00.516353 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.516327    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cadd9c-29823195-2w9qz"]
Sep 14 13:15:00.518141 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:15:00.518115    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e887453_6288_4733_9798_5d9fcfca0f0d.slice/crio-42e50d0aaf27280af6c6973320164f72ae620b96232d367795e6a0903341d552 WatchSource:0}: Error finding container 42e50d0aaf27280af6c6973320164f72ae620b96232d367795e6a0903341d552: Status 404 returned error can't find the container with id 42e50d0aaf27280af6c6973320164f72ae620b96232d367795e6a0903341d552
Sep 14 13:15:00.524571 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.524544    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc851d-29823195-t4bjm"]
Sep 14 13:15:00.525833 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:15:00.525809    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141cb3ed_d01b_403f_b518_80b393199f8d.slice/crio-81ba0f01cb570c318f254d6248c7a2ebff98284ee31791d03397a1e5131e759f WatchSource:0}: Error finding container 81ba0f01cb570c318f254d6248c7a2ebff98284ee31791d03397a1e5131e759f: Status 404 returned error can't find the container with id 81ba0f01cb570c318f254d6248c7a2ebff98284ee31791d03397a1e5131e759f
Sep 14 13:15:00.570762 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.570724    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-975227-29823195-rk8r7"]
Sep 14 13:15:00.571304 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:15:00.571283    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27855c4_c530_4798_b25a_b8bda3cbea9f.slice/crio-ae38591616582bf0e6f967125ffe80953d09724795600197afdaf72fce188459 WatchSource:0}: Error finding container ae38591616582bf0e6f967125ffe80953d09724795600197afdaf72fce188459: Status 404 returned error can't find the container with id ae38591616582bf0e6f967125ffe80953d09724795600197afdaf72fce188459
Sep 14 13:15:00.709613 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.709587    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"]
Sep 14 13:15:00.711548 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:15:00.711517    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7634a83e_5864_499a_9769_f29de26d5ad5.slice/crio-d928ac68919074c1423d2cdcba6402ef6fa61515367053344b3772117f0794f3 WatchSource:0}: Error finding container d928ac68919074c1423d2cdcba6402ef6fa61515367053344b3772117f0794f3: Status 404 returned error can't find the container with id d928ac68919074c1423d2cdcba6402ef6fa61515367053344b3772117f0794f3
Sep 14 13:15:00.782064 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.782038    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q" event={"ID":"7634a83e-5864-499a-9769-f29de26d5ad5","Type":"ContainerStarted","Data":"d928ac68919074c1423d2cdcba6402ef6fa61515367053344b3772117f0794f3"}
Sep 14 13:15:00.783371 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.783344    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc851d-29823195-t4bjm" event={"ID":"141cb3ed-d01b-403f-b518-80b393199f8d","Type":"ContainerStarted","Data":"81ba0f01cb570c318f254d6248c7a2ebff98284ee31791d03397a1e5131e759f"}
Sep 14 13:15:00.784572 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.784549    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-975227-29823195-rk8r7" event={"ID":"c27855c4-c530-4798-b25a-b8bda3cbea9f","Type":"ContainerStarted","Data":"ae38591616582bf0e6f967125ffe80953d09724795600197afdaf72fce188459"}
Sep 14 13:15:00.785670 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:00.785651    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cadd9c-29823195-2w9qz" event={"ID":"7e887453-6288-4733-9798-5d9fcfca0f0d","Type":"ContainerStarted","Data":"42e50d0aaf27280af6c6973320164f72ae620b96232d367795e6a0903341d552"}
Sep 14 13:15:01.792217 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:01.792187    2557 generic.go:356] "Generic (PLEG): container finished" podID="7e887453-6288-4733-9798-5d9fcfca0f0d" containerID="94eedaa0a0a867f52e755dca9157d07dd88b05f3eb630da02cf96debffa23695" exitCode=0
Sep 14 13:15:01.792561 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:01.792276    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cadd9c-29823195-2w9qz" event={"ID":"7e887453-6288-4733-9798-5d9fcfca0f0d","Type":"ContainerDied","Data":"94eedaa0a0a867f52e755dca9157d07dd88b05f3eb630da02cf96debffa23695"}
Sep 14 13:15:01.793968 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:01.793945    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q" event={"ID":"7634a83e-5864-499a-9769-f29de26d5ad5","Type":"ContainerStarted","Data":"bf5c04b05230857f22f2a5898dc6f7aa85c619352bfcab233d042b38a1995d02"}
Sep 14 13:15:01.795441 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:01.795421    2557 generic.go:356] "Generic (PLEG): container finished" podID="141cb3ed-d01b-403f-b518-80b393199f8d" containerID="cb7de97a703562397238af32f8f76ce1429d2a65cf5921f9f4edf6f31901f127" exitCode=0
Sep 14 13:15:01.795524 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:01.795495    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc851d-29823195-t4bjm" event={"ID":"141cb3ed-d01b-403f-b518-80b393199f8d","Type":"ContainerDied","Data":"cb7de97a703562397238af32f8f76ce1429d2a65cf5921f9f4edf6f31901f127"}
Sep 14 13:15:01.797050 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:01.797031    2557 generic.go:356] "Generic (PLEG): container finished" podID="c27855c4-c530-4798-b25a-b8bda3cbea9f" containerID="156fb9683365f7aa0f30fbc5fa75a073dda81cf1b439339613e5c0f55cb03fb9" exitCode=0
Sep 14 13:15:01.797136 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:01.797082    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-975227-29823195-rk8r7" event={"ID":"c27855c4-c530-4798-b25a-b8bda3cbea9f","Type":"ContainerDied","Data":"156fb9683365f7aa0f30fbc5fa75a073dda81cf1b439339613e5c0f55cb03fb9"}
Sep 14 13:15:01.847821 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:01.847714    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q" podStartSLOduration=1.84770098 podStartE2EDuration="1.84770098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:15:01.846594865 +0000 UTC m=+1020.281270384" watchObservedRunningTime="2026-09-14 13:15:01.84770098 +0000 UTC m=+1020.282376516"
Sep 14 13:15:02.939978 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:02.939949    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cadd9c-29823195-2w9qz"
Sep 14 13:15:02.997619 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:02.997595    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-975227-29823195-rk8r7"
Sep 14 13:15:03.000615 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:03.000587    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc851d-29823195-t4bjm"
Sep 14 13:15:03.809440 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:03.809406    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cadd9c-29823195-2w9qz"
Sep 14 13:15:03.809613 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:03.809439    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cadd9c-29823195-2w9qz" event={"ID":"7e887453-6288-4733-9798-5d9fcfca0f0d","Type":"ContainerDied","Data":"42e50d0aaf27280af6c6973320164f72ae620b96232d367795e6a0903341d552"}
Sep 14 13:15:03.809613 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:03.809493    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42e50d0aaf27280af6c6973320164f72ae620b96232d367795e6a0903341d552"
Sep 14 13:15:03.810916 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:03.810893    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc851d-29823195-t4bjm"
Sep 14 13:15:03.811066 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:03.810927    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc851d-29823195-t4bjm" event={"ID":"141cb3ed-d01b-403f-b518-80b393199f8d","Type":"ContainerDied","Data":"81ba0f01cb570c318f254d6248c7a2ebff98284ee31791d03397a1e5131e759f"}
Sep 14 13:15:03.811066 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:03.810963    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81ba0f01cb570c318f254d6248c7a2ebff98284ee31791d03397a1e5131e759f"
Sep 14 13:15:03.812243 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:03.812220    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-975227-29823195-rk8r7" event={"ID":"c27855c4-c530-4798-b25a-b8bda3cbea9f","Type":"ContainerDied","Data":"ae38591616582bf0e6f967125ffe80953d09724795600197afdaf72fce188459"}
Sep 14 13:15:03.812243 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:03.812243    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-975227-29823195-rk8r7"
Sep 14 13:15:03.812419 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:03.812258    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae38591616582bf0e6f967125ffe80953d09724795600197afdaf72fce188459"
Sep 14 13:15:07.348869 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:07.348836    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:15:07.349332 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:07.348898    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:15:07.361408 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:07.361385    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:15:07.843954 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:07.843922    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:15:10.425991 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:10.425953    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:10.426367 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:10.426003    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:10.438527 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:10.438503    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:10.862946 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:10.862918    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:15:16.082665 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:16.082631    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"]
Sep 14 13:15:16.083179 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:16.082897    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq" podUID="8110160b-b752-4f51-ab9a-a76e74aa919c" containerName="istio-proxy" containerID="cri-o://8aaa1b88d5c9b4fc3f93159ef70aca5c18548109a5b6c3777e6c17119457049f" gracePeriod=30
Sep 14 13:15:21.337732 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.337706    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:15:21.398033 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.397877    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-credential-socket\") pod \"8110160b-b752-4f51-ab9a-a76e74aa919c\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") "
Sep 14 13:15:21.398636 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.398246    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-workload-certs\") pod \"8110160b-b752-4f51-ab9a-a76e74aa919c\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") "
Sep 14 13:15:21.398636 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.398310    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k769s\" (UniqueName: \"kubernetes.io/projected/8110160b-b752-4f51-ab9a-a76e74aa919c-kube-api-access-k769s\") pod \"8110160b-b752-4f51-ab9a-a76e74aa919c\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") "
Sep 14 13:15:21.398636 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.398352    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8110160b-b752-4f51-ab9a-a76e74aa919c-istiod-ca-cert\") pod \"8110160b-b752-4f51-ab9a-a76e74aa919c\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") "
Sep 14 13:15:21.398636 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.398387    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-data\") pod \"8110160b-b752-4f51-ab9a-a76e74aa919c\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") "
Sep 14 13:15:21.398636 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.398432    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-podinfo\") pod \"8110160b-b752-4f51-ab9a-a76e74aa919c\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") "
Sep 14 13:15:21.398636 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.398443    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8110160b-b752-4f51-ab9a-a76e74aa919c" (UID: "8110160b-b752-4f51-ab9a-a76e74aa919c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:15:21.398636 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.398474    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-envoy\") pod \"8110160b-b752-4f51-ab9a-a76e74aa919c\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") "
Sep 14 13:15:21.398636 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.398541    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8110160b-b752-4f51-ab9a-a76e74aa919c" (UID: "8110160b-b752-4f51-ab9a-a76e74aa919c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:15:21.399122 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.398769    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8110160b-b752-4f51-ab9a-a76e74aa919c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8110160b-b752-4f51-ab9a-a76e74aa919c" (UID: "8110160b-b752-4f51-ab9a-a76e74aa919c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:15:21.399122 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.398810    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-token\") pod \"8110160b-b752-4f51-ab9a-a76e74aa919c\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") "
Sep 14 13:15:21.399122 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.398856    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-workload-socket\") pod \"8110160b-b752-4f51-ab9a-a76e74aa919c\" (UID: \"8110160b-b752-4f51-ab9a-a76e74aa919c\") "
Sep 14 13:15:21.399122 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.398993    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-data" (OuterVolumeSpecName: "istio-data") pod "8110160b-b752-4f51-ab9a-a76e74aa919c" (UID: "8110160b-b752-4f51-ab9a-a76e74aa919c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:15:21.399318 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.399266    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-workload-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:15:21.399318 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.399275    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8110160b-b752-4f51-ab9a-a76e74aa919c" (UID: "8110160b-b752-4f51-ab9a-a76e74aa919c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:15:21.399318 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.399284    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8110160b-b752-4f51-ab9a-a76e74aa919c-istiod-ca-cert\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:15:21.399478 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.399317    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-data\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:15:21.399478 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.399336    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-credential-socket\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:15:21.400930 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.400898    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8110160b-b752-4f51-ab9a-a76e74aa919c" (UID: "8110160b-b752-4f51-ab9a-a76e74aa919c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 13:15:21.401022 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.400963    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-token" (OuterVolumeSpecName: "istio-token") pod "8110160b-b752-4f51-ab9a-a76e74aa919c" (UID: "8110160b-b752-4f51-ab9a-a76e74aa919c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:15:21.401241 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.401212    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8110160b-b752-4f51-ab9a-a76e74aa919c-kube-api-access-k769s" (OuterVolumeSpecName: "kube-api-access-k769s") pod "8110160b-b752-4f51-ab9a-a76e74aa919c" (UID: "8110160b-b752-4f51-ab9a-a76e74aa919c"). InnerVolumeSpecName "kube-api-access-k769s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:15:21.401354 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.401276    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8110160b-b752-4f51-ab9a-a76e74aa919c" (UID: "8110160b-b752-4f51-ab9a-a76e74aa919c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:15:21.500255 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.500221    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-envoy\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:15:21.500255 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.500254    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-token\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:15:21.500458 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.500265    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8110160b-b752-4f51-ab9a-a76e74aa919c-workload-socket\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:15:21.500458 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.500279    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k769s\" (UniqueName: \"kubernetes.io/projected/8110160b-b752-4f51-ab9a-a76e74aa919c-kube-api-access-k769s\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:15:21.500458 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.500290    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8110160b-b752-4f51-ab9a-a76e74aa919c-istio-podinfo\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:15:21.921733 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.921703    2557 generic.go:356] "Generic (PLEG): container finished" podID="8110160b-b752-4f51-ab9a-a76e74aa919c" containerID="8aaa1b88d5c9b4fc3f93159ef70aca5c18548109a5b6c3777e6c17119457049f" exitCode=0
Sep 14 13:15:21.921964 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.921766    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq" event={"ID":"8110160b-b752-4f51-ab9a-a76e74aa919c","Type":"ContainerDied","Data":"8aaa1b88d5c9b4fc3f93159ef70aca5c18548109a5b6c3777e6c17119457049f"}
Sep 14 13:15:21.921964 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.921810    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq" event={"ID":"8110160b-b752-4f51-ab9a-a76e74aa919c","Type":"ContainerDied","Data":"7a0ec9354ea0514dae8257c57aa058e0466aba27eadaea24620f363a60ad280e"}
Sep 14 13:15:21.921964 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.921831    2557 scope.go:117] "RemoveContainer" containerID="8aaa1b88d5c9b4fc3f93159ef70aca5c18548109a5b6c3777e6c17119457049f"
Sep 14 13:15:21.921964 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.921766    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"
Sep 14 13:15:21.931725 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.931706    2557 scope.go:117] "RemoveContainer" containerID="23b2e7479beed2f36dac93743ea0121538d09128d38d791e4ed5495f5e2942ff"
Sep 14 13:15:21.939806 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.939786    2557 scope.go:117] "RemoveContainer" containerID="8aaa1b88d5c9b4fc3f93159ef70aca5c18548109a5b6c3777e6c17119457049f"
Sep 14 13:15:21.940070 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:15:21.940052    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aaa1b88d5c9b4fc3f93159ef70aca5c18548109a5b6c3777e6c17119457049f\": container with ID starting with 8aaa1b88d5c9b4fc3f93159ef70aca5c18548109a5b6c3777e6c17119457049f not found: ID does not exist" containerID="8aaa1b88d5c9b4fc3f93159ef70aca5c18548109a5b6c3777e6c17119457049f"
Sep 14 13:15:21.940133 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.940079    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aaa1b88d5c9b4fc3f93159ef70aca5c18548109a5b6c3777e6c17119457049f"} err="failed to get container status \"8aaa1b88d5c9b4fc3f93159ef70aca5c18548109a5b6c3777e6c17119457049f\": rpc error: code = NotFound desc = could not find container \"8aaa1b88d5c9b4fc3f93159ef70aca5c18548109a5b6c3777e6c17119457049f\": container with ID starting with 8aaa1b88d5c9b4fc3f93159ef70aca5c18548109a5b6c3777e6c17119457049f not found: ID does not exist"
Sep 14 13:15:21.940133 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.940095    2557 scope.go:117] "RemoveContainer" containerID="23b2e7479beed2f36dac93743ea0121538d09128d38d791e4ed5495f5e2942ff"
Sep 14 13:15:21.940333 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:15:21.940316    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23b2e7479beed2f36dac93743ea0121538d09128d38d791e4ed5495f5e2942ff\": container with ID starting with 23b2e7479beed2f36dac93743ea0121538d09128d38d791e4ed5495f5e2942ff not found: ID does not exist" containerID="23b2e7479beed2f36dac93743ea0121538d09128d38d791e4ed5495f5e2942ff"
Sep 14 13:15:21.940374 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.940340    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23b2e7479beed2f36dac93743ea0121538d09128d38d791e4ed5495f5e2942ff"} err="failed to get container status \"23b2e7479beed2f36dac93743ea0121538d09128d38d791e4ed5495f5e2942ff\": rpc error: code = NotFound desc = could not find container \"23b2e7479beed2f36dac93743ea0121538d09128d38d791e4ed5495f5e2942ff\": container with ID starting with 23b2e7479beed2f36dac93743ea0121538d09128d38d791e4ed5495f5e2942ff not found: ID does not exist"
Sep 14 13:15:21.946191 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.946172    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"]
Sep 14 13:15:21.954984 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:21.954957    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-c14a5e-openshift-default-685d7c8f76-8k6hq"]
Sep 14 13:15:22.050238 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:22.050202    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8110160b-b752-4f51-ab9a-a76e74aa919c" path="/var/lib/kubelet/pods/8110160b-b752-4f51-ab9a-a76e74aa919c/volumes"
Sep 14 13:15:24.361394 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.361360    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl"]
Sep 14 13:15:24.361987 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.361969    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8110160b-b752-4f51-ab9a-a76e74aa919c" containerName="istio-proxy"
Sep 14 13:15:24.362075 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.361989    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8110160b-b752-4f51-ab9a-a76e74aa919c" containerName="istio-proxy"
Sep 14 13:15:24.362075 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.362010    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c27855c4-c530-4798-b25a-b8bda3cbea9f" containerName="cleanup"
Sep 14 13:15:24.362075 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.362018    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c27855c4-c530-4798-b25a-b8bda3cbea9f" containerName="cleanup"
Sep 14 13:15:24.362075 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.362030    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e887453-6288-4733-9798-5d9fcfca0f0d" containerName="cleanup"
Sep 14 13:15:24.362075 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.362038    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e887453-6288-4733-9798-5d9fcfca0f0d" containerName="cleanup"
Sep 14 13:15:24.362075 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.362056    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="141cb3ed-d01b-403f-b518-80b393199f8d" containerName="cleanup"
Sep 14 13:15:24.362075 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.362064    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="141cb3ed-d01b-403f-b518-80b393199f8d" containerName="cleanup"
Sep 14 13:15:24.362428 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.362151    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8110160b-b752-4f51-ab9a-a76e74aa919c" containerName="istio-proxy"
Sep 14 13:15:24.362428 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.362163    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c27855c4-c530-4798-b25a-b8bda3cbea9f" containerName="cleanup"
Sep 14 13:15:24.362428 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.362175    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8110160b-b752-4f51-ab9a-a76e74aa919c" containerName="istio-proxy"
Sep 14 13:15:24.362428 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.362186    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e887453-6288-4733-9798-5d9fcfca0f0d" containerName="cleanup"
Sep 14 13:15:24.362428 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.362195    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="141cb3ed-d01b-403f-b518-80b393199f8d" containerName="cleanup"
Sep 14 13:15:24.366838 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.366817    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl"
Sep 14 13:15:24.372523 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.372500    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-ace1c6\""
Sep 14 13:15:24.386586 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.386553    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl"]
Sep 14 13:15:24.426286 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.426253    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77b8n\" (UniqueName: \"kubernetes.io/projected/04671737-f924-4fc0-a8e6-90f62e4944cd-kube-api-access-77b8n\") pod \"payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl\" (UID: \"04671737-f924-4fc0-a8e6-90f62e4944cd\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl"
Sep 14 13:15:24.426437 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.426329    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04671737-f924-4fc0-a8e6-90f62e4944cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl\" (UID: \"04671737-f924-4fc0-a8e6-90f62e4944cd\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl"
Sep 14 13:15:24.527507 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.527471    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77b8n\" (UniqueName: \"kubernetes.io/projected/04671737-f924-4fc0-a8e6-90f62e4944cd-kube-api-access-77b8n\") pod \"payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl\" (UID: \"04671737-f924-4fc0-a8e6-90f62e4944cd\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl"
Sep 14 13:15:24.527696 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.527564    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04671737-f924-4fc0-a8e6-90f62e4944cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl\" (UID: \"04671737-f924-4fc0-a8e6-90f62e4944cd\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl"
Sep 14 13:15:24.528214 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.528190    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04671737-f924-4fc0-a8e6-90f62e4944cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl\" (UID: \"04671737-f924-4fc0-a8e6-90f62e4944cd\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl"
Sep 14 13:15:24.536729 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.536700    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77b8n\" (UniqueName: \"kubernetes.io/projected/04671737-f924-4fc0-a8e6-90f62e4944cd-kube-api-access-77b8n\") pod \"payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl\" (UID: \"04671737-f924-4fc0-a8e6-90f62e4944cd\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl"
Sep 14 13:15:24.676805 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.676703    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl"
Sep 14 13:15:24.949581 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:24.949556    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl"]
Sep 14 13:15:24.952038 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:15:24.952009    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04671737_f924_4fc0_a8e6_90f62e4944cd.slice/crio-0ebafe3b5307a77572d2846122ecce284a09fe8d5a9599dec8be312e492195af WatchSource:0}: Error finding container 0ebafe3b5307a77572d2846122ecce284a09fe8d5a9599dec8be312e492195af: Status 404 returned error can't find the container with id 0ebafe3b5307a77572d2846122ecce284a09fe8d5a9599dec8be312e492195af
Sep 14 13:15:25.946463 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:25.946428    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl" event={"ID":"04671737-f924-4fc0-a8e6-90f62e4944cd","Type":"ContainerStarted","Data":"ae1af51c46798ddd9796d543cff81efd0e233da1c9bcfd43ca6c23ceba440d2c"}
Sep 14 13:15:25.946881 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:25.946486    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl" event={"ID":"04671737-f924-4fc0-a8e6-90f62e4944cd","Type":"ContainerStarted","Data":"0ebafe3b5307a77572d2846122ecce284a09fe8d5a9599dec8be312e492195af"}
Sep 14 13:15:25.946881 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:25.946511    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl"
Sep 14 13:15:25.968968 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:25.968921    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl" podStartSLOduration=1.968905516 podStartE2EDuration="1.968905516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:15:25.968529558 +0000 UTC m=+1044.403205077" watchObservedRunningTime="2026-09-14 13:15:25.968905516 +0000 UTC m=+1044.403581035"
Sep 14 13:15:34.110069 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.110032    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"]
Sep 14 13:15:34.110638 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.110623    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8110160b-b752-4f51-ab9a-a76e74aa919c" containerName="istio-proxy"
Sep 14 13:15:34.110684 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.110642    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8110160b-b752-4f51-ab9a-a76e74aa919c" containerName="istio-proxy"
Sep 14 13:15:34.114319 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.114301    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:34.118609 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.118584    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cadd9c\"/\"default-dockercfg-2l4n8\""
Sep 14 13:15:34.119099 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.118877    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cadd9c\"/\"kube-root-ca.crt\""
Sep 14 13:15:34.119099 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.118915    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cadd9c\"/\"test-model-cadd9c-kserve-self-signed-certs\""
Sep 14 13:15:34.119264 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.119147    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cadd9c\"/\"openshift-service-ca.crt\""
Sep 14 13:15:34.122509 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.122486    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"]
Sep 14 13:15:34.221428 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.221395    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-model-cache\") pod \"test-model-cadd9c-kserve-675546dfc7-kgqsq\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") " pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:34.221584 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.221442    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t52q4\" (UniqueName: \"kubernetes.io/projected/4af67e71-b311-45b8-8050-78c51f24ad1c-kube-api-access-t52q4\") pod \"test-model-cadd9c-kserve-675546dfc7-kgqsq\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") " pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:34.221584 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.221475    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-dshm\") pod \"test-model-cadd9c-kserve-675546dfc7-kgqsq\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") " pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:34.221584 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.221524    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4af67e71-b311-45b8-8050-78c51f24ad1c-tls-certs\") pod \"test-model-cadd9c-kserve-675546dfc7-kgqsq\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") " pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:34.221699 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.221615    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-home\") pod \"test-model-cadd9c-kserve-675546dfc7-kgqsq\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") " pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:34.221699 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.221644    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-tmp-dir\") pod \"test-model-cadd9c-kserve-675546dfc7-kgqsq\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") " pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:34.322878 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.322845    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4af67e71-b311-45b8-8050-78c51f24ad1c-tls-certs\") pod \"test-model-cadd9c-kserve-675546dfc7-kgqsq\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") " pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:34.323071 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.322902    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-home\") pod \"test-model-cadd9c-kserve-675546dfc7-kgqsq\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") " pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:34.323071 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.322930    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-tmp-dir\") pod \"test-model-cadd9c-kserve-675546dfc7-kgqsq\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") " pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:34.323071 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.322969    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-model-cache\") pod \"test-model-cadd9c-kserve-675546dfc7-kgqsq\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") " pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:34.323071 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.323010    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t52q4\" (UniqueName: \"kubernetes.io/projected/4af67e71-b311-45b8-8050-78c51f24ad1c-kube-api-access-t52q4\") pod \"test-model-cadd9c-kserve-675546dfc7-kgqsq\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") " pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:34.323071 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.323039    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-dshm\") pod \"test-model-cadd9c-kserve-675546dfc7-kgqsq\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") " pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:34.323449 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.323420    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-home\") pod \"test-model-cadd9c-kserve-675546dfc7-kgqsq\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") " pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:34.323573 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.323469    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-model-cache\") pod \"test-model-cadd9c-kserve-675546dfc7-kgqsq\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") " pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:34.323573 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.323513    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-tmp-dir\") pod \"test-model-cadd9c-kserve-675546dfc7-kgqsq\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") " pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:34.325807 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.325783    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-dshm\") pod \"test-model-cadd9c-kserve-675546dfc7-kgqsq\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") " pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:34.325992 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.325973    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4af67e71-b311-45b8-8050-78c51f24ad1c-tls-certs\") pod \"test-model-cadd9c-kserve-675546dfc7-kgqsq\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") " pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:34.332245 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.332224    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t52q4\" (UniqueName: \"kubernetes.io/projected/4af67e71-b311-45b8-8050-78c51f24ad1c-kube-api-access-t52q4\") pod \"test-model-cadd9c-kserve-675546dfc7-kgqsq\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") " pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:34.425354 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.425265    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:34.700315 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:34.700290    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"]
Sep 14 13:15:34.702969 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:15:34.702941    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4af67e71_b311_45b8_8050_78c51f24ad1c.slice/crio-2cab59a35f9f87ff275ea56a2dd658e70aca6339bf15fc8b81064c6b872ec900 WatchSource:0}: Error finding container 2cab59a35f9f87ff275ea56a2dd658e70aca6339bf15fc8b81064c6b872ec900: Status 404 returned error can't find the container with id 2cab59a35f9f87ff275ea56a2dd658e70aca6339bf15fc8b81064c6b872ec900
Sep 14 13:15:35.004927 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:35.004835    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq" event={"ID":"4af67e71-b311-45b8-8050-78c51f24ad1c","Type":"ContainerStarted","Data":"27b9c9187fc9e69425c0ffc42a526dd5e98e3bdbdf314160546ca4fcf4f4ba82"}
Sep 14 13:15:35.004927 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:35.004891    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq" event={"ID":"4af67e71-b311-45b8-8050-78c51f24ad1c","Type":"ContainerStarted","Data":"2cab59a35f9f87ff275ea56a2dd658e70aca6339bf15fc8b81064c6b872ec900"}
Sep 14 13:15:35.029182 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:35.029129    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq" podStartSLOduration=1.029114584 podStartE2EDuration="1.029114584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:15:35.025860624 +0000 UTC m=+1053.460536142" watchObservedRunningTime="2026-09-14 13:15:35.029114584 +0000 UTC m=+1053.463790102"
Sep 14 13:15:36.955808 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:36.955779    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl"
Sep 14 13:15:44.426085 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:44.426043    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:44.426460 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:44.426101    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:44.438395 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:44.438363    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:15:45.072858 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:15:45.072827    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:16:27.324362 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.324318    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"]
Sep 14 13:16:27.329925 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.329904    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.332991 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.332970    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-758ecaaa-openshift-default-dockercfg-9n5wd\""
Sep 14 13:16:27.339962 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.339939    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"]
Sep 14 13:16:27.498857 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.498829    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a02d0d5-539f-493b-8798-50054df800a3-istio-token\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.499053 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.498872    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a02d0d5-539f-493b-8798-50054df800a3-istio-podinfo\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.499053 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.498894    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-istio-envoy\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.499053 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.498994    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-workload-socket\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.499053 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.499030    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-workload-certs\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.499238 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.499055    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a02d0d5-539f-493b-8798-50054df800a3-istiod-ca-cert\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.499238 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.499105    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-istio-data\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.499238 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.499170    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtxtx\" (UniqueName: \"kubernetes.io/projected/3a02d0d5-539f-493b-8798-50054df800a3-kube-api-access-gtxtx\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.499238 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.499228    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-credential-socket\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.600170 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.600099    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-workload-socket\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.600170 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.600129    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-workload-certs\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.600170 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.600147    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a02d0d5-539f-493b-8798-50054df800a3-istiod-ca-cert\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.600170 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.600167    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-istio-data\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.600489 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.600225    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtxtx\" (UniqueName: \"kubernetes.io/projected/3a02d0d5-539f-493b-8798-50054df800a3-kube-api-access-gtxtx\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.600489 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.600268    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-credential-socket\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.600489 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.600311    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a02d0d5-539f-493b-8798-50054df800a3-istio-token\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.600489 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.600350    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a02d0d5-539f-493b-8798-50054df800a3-istio-podinfo\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.600489 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.600377    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-istio-envoy\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.600797 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.600506    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-workload-socket\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.600797 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.600555    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-workload-certs\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.600923 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.600833    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-credential-socket\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.601021 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.601002    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a02d0d5-539f-493b-8798-50054df800a3-istiod-ca-cert\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.601087 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.601069    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-istio-data\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.603341 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.603319    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-istio-envoy\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.603472 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.603456    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a02d0d5-539f-493b-8798-50054df800a3-istio-podinfo\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.609165 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.609142    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a02d0d5-539f-493b-8798-50054df800a3-istio-token\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.609387 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.609370    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtxtx\" (UniqueName: \"kubernetes.io/projected/3a02d0d5-539f-493b-8798-50054df800a3-kube-api-access-gtxtx\") pod \"e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") " pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.641978 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.641954    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:27.945635 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.945611    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"]
Sep 14 13:16:27.948021 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:16:27.947992    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a02d0d5_539f_493b_8798_50054df800a3.slice/crio-8654c14def55e4ec0c49903bb6d9f94ab14a0ef6ab1abc44dc150ceed4cf5f9d WatchSource:0}: Error finding container 8654c14def55e4ec0c49903bb6d9f94ab14a0ef6ab1abc44dc150ceed4cf5f9d: Status 404 returned error can't find the container with id 8654c14def55e4ec0c49903bb6d9f94ab14a0ef6ab1abc44dc150ceed4cf5f9d
Sep 14 13:16:27.959074 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.959027    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:16:27.959198 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.959101    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:16:27.959198 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:27.959131    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:16:28.297633 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:28.297547    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm" event={"ID":"3a02d0d5-539f-493b-8798-50054df800a3","Type":"ContainerStarted","Data":"da75bad4cc9599e9eff55b0458640a7e8eb47473d1d1353ba4ded238773205f0"}
Sep 14 13:16:28.297633 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:28.297595    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm" event={"ID":"3a02d0d5-539f-493b-8798-50054df800a3","Type":"ContainerStarted","Data":"8654c14def55e4ec0c49903bb6d9f94ab14a0ef6ab1abc44dc150ceed4cf5f9d"}
Sep 14 13:16:28.321360 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:28.321313    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm" podStartSLOduration=1.321299193 podStartE2EDuration="1.321299193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:16:28.320162623 +0000 UTC m=+1106.754838142" watchObservedRunningTime="2026-09-14 13:16:28.321299193 +0000 UTC m=+1106.755974711"
Sep 14 13:16:28.642316 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:28.642238    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:29.647258 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:29.647225    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:30.308635 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:30.308595    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:30.309598 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:30.309580    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:37.553324 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:37.553288    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25"]
Sep 14 13:16:37.557020 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:37.556999    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25"
Sep 14 13:16:37.559632 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:37.559607    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-758ecaaa-dockercfg-wdd6b\""
Sep 14 13:16:37.559919 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:37.559904    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-758ecaaa\""
Sep 14 13:16:37.565388 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:37.565303    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25"]
Sep 14 13:16:37.591162 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:37.591138    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2"]
Sep 14 13:16:37.592503 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:37.592483    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpbvd\" (UniqueName: \"kubernetes.io/projected/62c9a08b-1343-459f-826c-0d55e0ef9b08-kube-api-access-dpbvd\") pod \"payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25\" (UID: \"62c9a08b-1343-459f-826c-0d55e0ef9b08\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25"
Sep 14 13:16:37.592563 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:37.592520    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c9a08b-1343-459f-826c-0d55e0ef9b08-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25\" (UID: \"62c9a08b-1343-459f-826c-0d55e0ef9b08\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25"
Sep 14 13:16:37.594498 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:37.594484    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2"
Sep 14 13:16:37.603189 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:37.603162    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2"]
Sep 14 13:16:37.693657 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:37.693621    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a007d7c-fc16-4c28-b147-35173882b950-plugins-config-volume\") pod \"payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2\" (UID: \"0a007d7c-fc16-4c28-b147-35173882b950\") " pod="openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2"
Sep 14 13:16:37.693835 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:37.693680    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9sw2\" (UniqueName: \"kubernetes.io/projected/0a007d7c-fc16-4c28-b147-35173882b950-kube-api-access-h9sw2\") pod \"payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2\" (UID: \"0a007d7c-fc16-4c28-b147-35173882b950\") " pod="openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2"
Sep 14 13:16:37.693835 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:37.693763    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpbvd\" (UniqueName: \"kubernetes.io/projected/62c9a08b-1343-459f-826c-0d55e0ef9b08-kube-api-access-dpbvd\") pod \"payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25\" (UID: \"62c9a08b-1343-459f-826c-0d55e0ef9b08\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25"
Sep 14 13:16:37.693835 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:37.693807    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c9a08b-1343-459f-826c-0d55e0ef9b08-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25\" (UID: \"62c9a08b-1343-459f-826c-0d55e0ef9b08\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25"
Sep 14 13:16:37.694526 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:37.694509    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c9a08b-1343-459f-826c-0d55e0ef9b08-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25\" (UID: \"62c9a08b-1343-459f-826c-0d55e0ef9b08\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25"
Sep 14 13:16:37.702145 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:37.702123    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpbvd\" (UniqueName: \"kubernetes.io/projected/62c9a08b-1343-459f-826c-0d55e0ef9b08-kube-api-access-dpbvd\") pod \"payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25\" (UID: \"62c9a08b-1343-459f-826c-0d55e0ef9b08\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25"
Sep 14 13:16:37.794908 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:37.794871    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a007d7c-fc16-4c28-b147-35173882b950-plugins-config-volume\") pod \"payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2\" (UID: \"0a007d7c-fc16-4c28-b147-35173882b950\") " pod="openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2"
Sep 14 13:16:37.795073 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:37.794925    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9sw2\" (UniqueName: \"kubernetes.io/projected/0a007d7c-fc16-4c28-b147-35173882b950-kube-api-access-h9sw2\") pod \"payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2\" (UID: \"0a007d7c-fc16-4c28-b147-35173882b950\") " pod="openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2"
Sep 14 13:16:37.795553 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:37.795534    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a007d7c-fc16-4c28-b147-35173882b950-plugins-config-volume\") pod \"payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2\" (UID: \"0a007d7c-fc16-4c28-b147-35173882b950\") " pod="openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2"
Sep 14 13:16:37.803793 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:37.803707    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9sw2\" (UniqueName: \"kubernetes.io/projected/0a007d7c-fc16-4c28-b147-35173882b950-kube-api-access-h9sw2\") pod \"payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2\" (UID: \"0a007d7c-fc16-4c28-b147-35173882b950\") " pod="openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2"
Sep 14 13:16:37.869091 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:37.869064    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25"
Sep 14 13:16:37.905915 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:37.905883    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2"
Sep 14 13:16:38.142189 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:38.142166    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25"]
Sep 14 13:16:38.144288 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:16:38.144261    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c9a08b_1343_459f_826c_0d55e0ef9b08.slice/crio-58b88aca02b6aa5aee40d9c9cbd35283815576aa02948a16e8120fdef037ca69 WatchSource:0}: Error finding container 58b88aca02b6aa5aee40d9c9cbd35283815576aa02948a16e8120fdef037ca69: Status 404 returned error can't find the container with id 58b88aca02b6aa5aee40d9c9cbd35283815576aa02948a16e8120fdef037ca69
Sep 14 13:16:38.194794 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:16:38.194764    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a007d7c_fc16_4c28_b147_35173882b950.slice/crio-c4d1dfbf4c4ebf41f613450582879aa7187b3d0371e5858486728a14a1fdfa29 WatchSource:0}: Error finding container c4d1dfbf4c4ebf41f613450582879aa7187b3d0371e5858486728a14a1fdfa29: Status 404 returned error can't find the container with id c4d1dfbf4c4ebf41f613450582879aa7187b3d0371e5858486728a14a1fdfa29
Sep 14 13:16:38.195037 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:38.195010    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2"]
Sep 14 13:16:38.357538 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:38.356826    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25" event={"ID":"62c9a08b-1343-459f-826c-0d55e0ef9b08","Type":"ContainerStarted","Data":"25aa97eb2ed20f6fb6671671f463186255e79425f4902eed49a8035193bd3bd1"}
Sep 14 13:16:38.357538 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:38.356901    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25" event={"ID":"62c9a08b-1343-459f-826c-0d55e0ef9b08","Type":"ContainerStarted","Data":"58b88aca02b6aa5aee40d9c9cbd35283815576aa02948a16e8120fdef037ca69"}
Sep 14 13:16:38.357538 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:38.356950    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25"
Sep 14 13:16:38.358869 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:38.358844    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2" event={"ID":"0a007d7c-fc16-4c28-b147-35173882b950","Type":"ContainerStarted","Data":"c1b0d44442493e1745ccdb09307e49d6232c1d23c1fd2849bb0108b0bbe62735"}
Sep 14 13:16:38.358997 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:38.358879    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2" event={"ID":"0a007d7c-fc16-4c28-b147-35173882b950","Type":"ContainerStarted","Data":"c4d1dfbf4c4ebf41f613450582879aa7187b3d0371e5858486728a14a1fdfa29"}
Sep 14 13:16:38.358997 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:38.358907    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2"
Sep 14 13:16:38.373952 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:38.373905    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25" podStartSLOduration=1.373887851 podStartE2EDuration="1.373887851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:16:38.371370793 +0000 UTC m=+1116.806046313" watchObservedRunningTime="2026-09-14 13:16:38.373887851 +0000 UTC m=+1116.808563374"
Sep 14 13:16:38.390258 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:38.390221    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2" podStartSLOduration=1.3902082980000001 podStartE2EDuration="1.390208298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:16:38.387775132 +0000 UTC m=+1116.822450654" watchObservedRunningTime="2026-09-14 13:16:38.390208298 +0000 UTC m=+1116.824883817"
Sep 14 13:16:49.367298 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:49.367265    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25"
Sep 14 13:16:49.367707 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:49.367347    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2"
Sep 14 13:16:58.476241 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:58.476156    2557 generic.go:356] "Generic (PLEG): container finished" podID="3a02d0d5-539f-493b-8798-50054df800a3" containerID="da75bad4cc9599e9eff55b0458640a7e8eb47473d1d1353ba4ded238773205f0" exitCode=0
Sep 14 13:16:58.476639 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:58.476231    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm" event={"ID":"3a02d0d5-539f-493b-8798-50054df800a3","Type":"ContainerDied","Data":"da75bad4cc9599e9eff55b0458640a7e8eb47473d1d1353ba4ded238773205f0"}
Sep 14 13:16:58.476691 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:58.476651    2557 scope.go:117] "RemoveContainer" containerID="da75bad4cc9599e9eff55b0458640a7e8eb47473d1d1353ba4ded238773205f0"
Sep 14 13:16:58.477815 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:58.477790    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:16:58.477875 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:58.477854    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:16:58.477914 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:58.477886    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:16:59.484154 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:59.484122    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm" event={"ID":"3a02d0d5-539f-493b-8798-50054df800a3","Type":"ContainerStarted","Data":"9fb3f669aba50e8606df3b3dfa27641bb62ff7420d045112c0c99c4d238df0ba"}
Sep 14 13:16:59.642905 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:59.642869    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:16:59.644092 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:59.644069    2557 patch_prober.go:28] interesting pod/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 14 13:16:59.644213 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:16:59.644117    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm" podUID="3a02d0d5-539f-493b-8798-50054df800a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 14 13:17:00.309520 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:00.309491    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:17:00.642469 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:00.642390    2557 patch_prober.go:28] interesting pod/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 14 13:17:00.642469 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:00.642447    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm" podUID="3a02d0d5-539f-493b-8798-50054df800a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 14 13:17:01.642844 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:01.642807    2557 patch_prober.go:28] interesting pod/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 14 13:17:01.643211 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:01.642864    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm" podUID="3a02d0d5-539f-493b-8798-50054df800a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 14 13:17:02.642536 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:02.642504    2557 patch_prober.go:28] interesting pod/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 14 13:17:02.642705 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:02.642567    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm" podUID="3a02d0d5-539f-493b-8798-50054df800a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 14 13:17:03.642415 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:03.642372    2557 patch_prober.go:28] interesting pod/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 14 13:17:03.642823 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:03.642442    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm" podUID="3a02d0d5-539f-493b-8798-50054df800a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 14 13:17:04.646885 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:04.646851    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:17:04.648045 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:04.648024    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:17:10.678093 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:10.678058    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-758ecaaa-2sd65"]
Sep 14 13:17:10.682486 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:10.682461    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-758ecaaa-2sd65"
Sep 14 13:17:10.685491 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:10.685286    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbzp\""
Sep 14 13:17:10.686984 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:10.686959    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-758ecaaa-2sd65"]
Sep 14 13:17:10.787577 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:10.787549    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c3dede-f478-4184-9469-62a376ba9218-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-758ecaaa-2sd65\" (UID: \"79c3dede-f478-4184-9469-62a376ba9218\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-758ecaaa-2sd65"
Sep 14 13:17:10.888829 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:10.888804    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c3dede-f478-4184-9469-62a376ba9218-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-758ecaaa-2sd65\" (UID: \"79c3dede-f478-4184-9469-62a376ba9218\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-758ecaaa-2sd65"
Sep 14 13:17:10.889408 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:10.889386    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c3dede-f478-4184-9469-62a376ba9218-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-758ecaaa-2sd65\" (UID: \"79c3dede-f478-4184-9469-62a376ba9218\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-758ecaaa-2sd65"
Sep 14 13:17:10.994009 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:10.993956    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-758ecaaa-2sd65"
Sep 14 13:17:11.267693 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:11.267667    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-758ecaaa-2sd65"]
Sep 14 13:17:11.269333 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:17:11.269307    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c3dede_f478_4184_9469_62a376ba9218.slice/crio-ccdba5be33b984dc5384442bcd83d203e868c0b2db35c25f3d350cda40f39095 WatchSource:0}: Error finding container ccdba5be33b984dc5384442bcd83d203e868c0b2db35c25f3d350cda40f39095: Status 404 returned error can't find the container with id ccdba5be33b984dc5384442bcd83d203e868c0b2db35c25f3d350cda40f39095
Sep 14 13:17:11.556780 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:11.556688    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-758ecaaa-2sd65" event={"ID":"79c3dede-f478-4184-9469-62a376ba9218","Type":"ContainerStarted","Data":"ccdba5be33b984dc5384442bcd83d203e868c0b2db35c25f3d350cda40f39095"}
Sep 14 13:17:12.564755 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:12.564717    2557 generic.go:356] "Generic (PLEG): container finished" podID="79c3dede-f478-4184-9469-62a376ba9218" containerID="1bacffdbe7c0e5cbbeba637687dfbdbc93b2670a772e85ec03366a309144d7dd" exitCode=0
Sep 14 13:17:12.565179 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:12.564814    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-758ecaaa-2sd65" event={"ID":"79c3dede-f478-4184-9469-62a376ba9218","Type":"ContainerDied","Data":"1bacffdbe7c0e5cbbeba637687dfbdbc93b2670a772e85ec03366a309144d7dd"}
Sep 14 13:17:13.712812 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:13.712782    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-758ecaaa-2sd65"
Sep 14 13:17:13.819264 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:13.819233    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c3dede-f478-4184-9469-62a376ba9218-maas-api-service-ca\") pod \"79c3dede-f478-4184-9469-62a376ba9218\" (UID: \"79c3dede-f478-4184-9469-62a376ba9218\") "
Sep 14 13:17:13.819561 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:13.819537    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c3dede-f478-4184-9469-62a376ba9218-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "79c3dede-f478-4184-9469-62a376ba9218" (UID: "79c3dede-f478-4184-9469-62a376ba9218"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:17:13.920801 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:13.920705    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c3dede-f478-4184-9469-62a376ba9218-maas-api-service-ca\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:17:14.579869 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:14.579829    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-758ecaaa-2sd65" event={"ID":"79c3dede-f478-4184-9469-62a376ba9218","Type":"ContainerDied","Data":"ccdba5be33b984dc5384442bcd83d203e868c0b2db35c25f3d350cda40f39095"}
Sep 14 13:17:14.580034 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:14.579913    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccdba5be33b984dc5384442bcd83d203e868c0b2db35c25f3d350cda40f39095"
Sep 14 13:17:14.580034 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:14.579843    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-758ecaaa-2sd65"
Sep 14 13:17:21.198367 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.198246    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2"]
Sep 14 13:17:21.198879 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.198512    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2" podUID="0a007d7c-fc16-4c28-b147-35173882b950" containerName="payload-processing" containerID="cri-o://c1b0d44442493e1745ccdb09307e49d6232c1d23c1fd2849bb0108b0bbe62735" gracePeriod=30
Sep 14 13:17:21.201839 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.201813    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25"]
Sep 14 13:17:21.202210 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.202167    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25" podUID="62c9a08b-1343-459f-826c-0d55e0ef9b08" containerName="payload-pre-processing" containerID="cri-o://25aa97eb2ed20f6fb6671671f463186255e79425f4902eed49a8035193bd3bd1" gracePeriod=30
Sep 14 13:17:21.363870 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.363809    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-758ecaaa-2sd65"]
Sep 14 13:17:21.365679 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.365651    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-758ecaaa-2sd65"]
Sep 14 13:17:21.497326 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.497285    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2"
Sep 14 13:17:21.500243 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.500223    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25"
Sep 14 13:17:21.594084 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.594049    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9sw2\" (UniqueName: \"kubernetes.io/projected/0a007d7c-fc16-4c28-b147-35173882b950-kube-api-access-h9sw2\") pod \"0a007d7c-fc16-4c28-b147-35173882b950\" (UID: \"0a007d7c-fc16-4c28-b147-35173882b950\") "
Sep 14 13:17:21.594265 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.594105    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpbvd\" (UniqueName: \"kubernetes.io/projected/62c9a08b-1343-459f-826c-0d55e0ef9b08-kube-api-access-dpbvd\") pod \"62c9a08b-1343-459f-826c-0d55e0ef9b08\" (UID: \"62c9a08b-1343-459f-826c-0d55e0ef9b08\") "
Sep 14 13:17:21.594265 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.594173    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a007d7c-fc16-4c28-b147-35173882b950-plugins-config-volume\") pod \"0a007d7c-fc16-4c28-b147-35173882b950\" (UID: \"0a007d7c-fc16-4c28-b147-35173882b950\") "
Sep 14 13:17:21.594265 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.594215    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c9a08b-1343-459f-826c-0d55e0ef9b08-plugins-config-volume\") pod \"62c9a08b-1343-459f-826c-0d55e0ef9b08\" (UID: \"62c9a08b-1343-459f-826c-0d55e0ef9b08\") "
Sep 14 13:17:21.594866 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.594798    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a007d7c-fc16-4c28-b147-35173882b950-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0a007d7c-fc16-4c28-b147-35173882b950" (UID: "0a007d7c-fc16-4c28-b147-35173882b950"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:17:21.594987 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.594954    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c9a08b-1343-459f-826c-0d55e0ef9b08-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "62c9a08b-1343-459f-826c-0d55e0ef9b08" (UID: "62c9a08b-1343-459f-826c-0d55e0ef9b08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:17:21.596483 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.596459    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c9a08b-1343-459f-826c-0d55e0ef9b08-kube-api-access-dpbvd" (OuterVolumeSpecName: "kube-api-access-dpbvd") pod "62c9a08b-1343-459f-826c-0d55e0ef9b08" (UID: "62c9a08b-1343-459f-826c-0d55e0ef9b08"). InnerVolumeSpecName "kube-api-access-dpbvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:17:21.596784 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.596752    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a007d7c-fc16-4c28-b147-35173882b950-kube-api-access-h9sw2" (OuterVolumeSpecName: "kube-api-access-h9sw2") pod "0a007d7c-fc16-4c28-b147-35173882b950" (UID: "0a007d7c-fc16-4c28-b147-35173882b950"). InnerVolumeSpecName "kube-api-access-h9sw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:17:21.621355 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.621320    2557 generic.go:356] "Generic (PLEG): container finished" podID="0a007d7c-fc16-4c28-b147-35173882b950" containerID="c1b0d44442493e1745ccdb09307e49d6232c1d23c1fd2849bb0108b0bbe62735" exitCode=0
Sep 14 13:17:21.621487 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.621363    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2"
Sep 14 13:17:21.621487 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.621360    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2" event={"ID":"0a007d7c-fc16-4c28-b147-35173882b950","Type":"ContainerDied","Data":"c1b0d44442493e1745ccdb09307e49d6232c1d23c1fd2849bb0108b0bbe62735"}
Sep 14 13:17:21.621487 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.621482    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2" event={"ID":"0a007d7c-fc16-4c28-b147-35173882b950","Type":"ContainerDied","Data":"c4d1dfbf4c4ebf41f613450582879aa7187b3d0371e5858486728a14a1fdfa29"}
Sep 14 13:17:21.621647 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.621505    2557 scope.go:117] "RemoveContainer" containerID="c1b0d44442493e1745ccdb09307e49d6232c1d23c1fd2849bb0108b0bbe62735"
Sep 14 13:17:21.623058 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.623028    2557 generic.go:356] "Generic (PLEG): container finished" podID="62c9a08b-1343-459f-826c-0d55e0ef9b08" containerID="25aa97eb2ed20f6fb6671671f463186255e79425f4902eed49a8035193bd3bd1" exitCode=0
Sep 14 13:17:21.623182 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.623059    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25"
Sep 14 13:17:21.623182 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.623084    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25" event={"ID":"62c9a08b-1343-459f-826c-0d55e0ef9b08","Type":"ContainerDied","Data":"25aa97eb2ed20f6fb6671671f463186255e79425f4902eed49a8035193bd3bd1"}
Sep 14 13:17:21.623182 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.623126    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25" event={"ID":"62c9a08b-1343-459f-826c-0d55e0ef9b08","Type":"ContainerDied","Data":"58b88aca02b6aa5aee40d9c9cbd35283815576aa02948a16e8120fdef037ca69"}
Sep 14 13:17:21.631709 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.631660    2557 scope.go:117] "RemoveContainer" containerID="c1b0d44442493e1745ccdb09307e49d6232c1d23c1fd2849bb0108b0bbe62735"
Sep 14 13:17:21.632081 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:17:21.632058    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b0d44442493e1745ccdb09307e49d6232c1d23c1fd2849bb0108b0bbe62735\": container with ID starting with c1b0d44442493e1745ccdb09307e49d6232c1d23c1fd2849bb0108b0bbe62735 not found: ID does not exist" containerID="c1b0d44442493e1745ccdb09307e49d6232c1d23c1fd2849bb0108b0bbe62735"
Sep 14 13:17:21.632143 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.632093    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b0d44442493e1745ccdb09307e49d6232c1d23c1fd2849bb0108b0bbe62735"} err="failed to get container status \"c1b0d44442493e1745ccdb09307e49d6232c1d23c1fd2849bb0108b0bbe62735\": rpc error: code = NotFound desc = could not find container \"c1b0d44442493e1745ccdb09307e49d6232c1d23c1fd2849bb0108b0bbe62735\": container with ID starting with c1b0d44442493e1745ccdb09307e49d6232c1d23c1fd2849bb0108b0bbe62735 not found: ID does not exist"
Sep 14 13:17:21.632143 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.632117    2557 scope.go:117] "RemoveContainer" containerID="25aa97eb2ed20f6fb6671671f463186255e79425f4902eed49a8035193bd3bd1"
Sep 14 13:17:21.642338 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.642324    2557 scope.go:117] "RemoveContainer" containerID="25aa97eb2ed20f6fb6671671f463186255e79425f4902eed49a8035193bd3bd1"
Sep 14 13:17:21.642593 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:17:21.642575    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25aa97eb2ed20f6fb6671671f463186255e79425f4902eed49a8035193bd3bd1\": container with ID starting with 25aa97eb2ed20f6fb6671671f463186255e79425f4902eed49a8035193bd3bd1 not found: ID does not exist" containerID="25aa97eb2ed20f6fb6671671f463186255e79425f4902eed49a8035193bd3bd1"
Sep 14 13:17:21.642660 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.642600    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25aa97eb2ed20f6fb6671671f463186255e79425f4902eed49a8035193bd3bd1"} err="failed to get container status \"25aa97eb2ed20f6fb6671671f463186255e79425f4902eed49a8035193bd3bd1\": rpc error: code = NotFound desc = could not find container \"25aa97eb2ed20f6fb6671671f463186255e79425f4902eed49a8035193bd3bd1\": container with ID starting with 25aa97eb2ed20f6fb6671671f463186255e79425f4902eed49a8035193bd3bd1 not found: ID does not exist"
Sep 14 13:17:21.647936 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.647912    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25"]
Sep 14 13:17:21.652665 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.652628    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-758ecaaa-5d4f695698-gtf25"]
Sep 14 13:17:21.681496 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.681437    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2"]
Sep 14 13:17:21.687095 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.687057    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-758ecaaa-77bdd8df7c-mqps2"]
Sep 14 13:17:21.695448 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.695418    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a007d7c-fc16-4c28-b147-35173882b950-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:17:21.695567 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.695453    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c9a08b-1343-459f-826c-0d55e0ef9b08-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:17:21.695567 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.695484    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9sw2\" (UniqueName: \"kubernetes.io/projected/0a007d7c-fc16-4c28-b147-35173882b950-kube-api-access-h9sw2\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:17:21.695567 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:21.695501    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpbvd\" (UniqueName: \"kubernetes.io/projected/62c9a08b-1343-459f-826c-0d55e0ef9b08-kube-api-access-dpbvd\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:17:22.050364 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:22.050333    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a007d7c-fc16-4c28-b147-35173882b950" path="/var/lib/kubelet/pods/0a007d7c-fc16-4c28-b147-35173882b950/volumes"
Sep 14 13:17:22.050672 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:22.050658    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62c9a08b-1343-459f-826c-0d55e0ef9b08" path="/var/lib/kubelet/pods/62c9a08b-1343-459f-826c-0d55e0ef9b08/volumes"
Sep 14 13:17:22.050965 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:22.050953    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79c3dede-f478-4184-9469-62a376ba9218" path="/var/lib/kubelet/pods/79c3dede-f478-4184-9469-62a376ba9218/volumes"
Sep 14 13:17:27.913221 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:27.913137    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"]
Sep 14 13:17:27.913725 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:27.913707    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62c9a08b-1343-459f-826c-0d55e0ef9b08" containerName="payload-pre-processing"
Sep 14 13:17:27.913864 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:27.913729    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="62c9a08b-1343-459f-826c-0d55e0ef9b08" containerName="payload-pre-processing"
Sep 14 13:17:27.913864 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:27.913765    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a007d7c-fc16-4c28-b147-35173882b950" containerName="payload-processing"
Sep 14 13:17:27.913864 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:27.913773    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a007d7c-fc16-4c28-b147-35173882b950" containerName="payload-processing"
Sep 14 13:17:27.913864 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:27.913795    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79c3dede-f478-4184-9469-62a376ba9218" containerName="revoke-keys"
Sep 14 13:17:27.913864 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:27.913804    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="79c3dede-f478-4184-9469-62a376ba9218" containerName="revoke-keys"
Sep 14 13:17:27.914119 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:27.913900    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="62c9a08b-1343-459f-826c-0d55e0ef9b08" containerName="payload-pre-processing"
Sep 14 13:17:27.914119 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:27.913913    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a007d7c-fc16-4c28-b147-35173882b950" containerName="payload-processing"
Sep 14 13:17:27.914119 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:27.913927    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="79c3dede-f478-4184-9469-62a376ba9218" containerName="revoke-keys"
Sep 14 13:17:27.918571 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:27.918549    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:27.921819 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:27.921798    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b3d6c5\"/\"e2e-conflict-a-c95ce5f6-kserve-self-signed-certs\""
Sep 14 13:17:27.931197 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:27.931176    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"]
Sep 14 13:17:27.952628 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:27.952591    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-tls-certs\") pod \"e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") " pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:27.952765 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:27.952639    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwnx9\" (UniqueName: \"kubernetes.io/projected/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-kube-api-access-mwnx9\") pod \"e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") " pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:27.952765 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:27.952666    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-tmp-dir\") pod \"e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") " pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:27.952900 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:27.952762    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-dshm\") pod \"e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") " pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:27.952900 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:27.952804    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-home\") pod \"e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") " pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:27.952900 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:27.952853    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-model-cache\") pod \"e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") " pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:28.053693 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:28.053664    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-dshm\") pod \"e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") " pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:28.053693 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:28.053699    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-home\") pod \"e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") " pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:28.053943 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:28.053732    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-model-cache\") pod \"e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") " pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:28.053943 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:28.053801    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-tls-certs\") pod \"e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") " pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:28.053943 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:28.053817    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwnx9\" (UniqueName: \"kubernetes.io/projected/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-kube-api-access-mwnx9\") pod \"e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") " pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:28.053943 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:28.053837    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-tmp-dir\") pod \"e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") " pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:28.054185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:28.054123    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-home\") pod \"e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") " pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:28.054248 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:28.054236    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-model-cache\") pod \"e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") " pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:28.054385 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:28.054365    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-tmp-dir\") pod \"e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") " pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:28.056088 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:28.056061    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-dshm\") pod \"e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") " pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:28.056379 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:28.056361    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-tls-certs\") pod \"e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") " pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:28.062142 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:28.062122    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwnx9\" (UniqueName: \"kubernetes.io/projected/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-kube-api-access-mwnx9\") pod \"e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") " pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:28.230275 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:28.230251    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:28.511888 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:28.511860    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"]
Sep 14 13:17:28.514644 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:17:28.514616    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ee1ffa7_a413_4af3_abe9_1fe91742a1ba.slice/crio-2866660ade895a7d016869dbd6008d33b00fd272505d7d4da925477e313feb6a WatchSource:0}: Error finding container 2866660ade895a7d016869dbd6008d33b00fd272505d7d4da925477e313feb6a: Status 404 returned error can't find the container with id 2866660ade895a7d016869dbd6008d33b00fd272505d7d4da925477e313feb6a
Sep 14 13:17:28.517167 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:28.517144    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"]
Sep 14 13:17:28.517409 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:28.517388    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm" podUID="3a02d0d5-539f-493b-8798-50054df800a3" containerName="istio-proxy" containerID="cri-o://9fb3f669aba50e8606df3b3dfa27641bb62ff7420d045112c0c99c4d238df0ba" gracePeriod=30
Sep 14 13:17:28.668306 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:28.668264    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v" event={"ID":"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba","Type":"ContainerStarted","Data":"430547208d9810800417a9756d04b6607cb5548c1f2ed3a5ada32019dc6fac0f"}
Sep 14 13:17:28.668457 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:28.668317    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v" event={"ID":"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba","Type":"ContainerStarted","Data":"2866660ade895a7d016869dbd6008d33b00fd272505d7d4da925477e313feb6a"}
Sep 14 13:17:28.689292 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:28.689242    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v" podStartSLOduration=1.689228064 podStartE2EDuration="1.689228064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:17:28.687185349 +0000 UTC m=+1167.121860868" watchObservedRunningTime="2026-09-14 13:17:28.689228064 +0000 UTC m=+1167.123903583"
Sep 14 13:17:33.697107 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.697076    2557 generic.go:356] "Generic (PLEG): container finished" podID="3a02d0d5-539f-493b-8798-50054df800a3" containerID="9fb3f669aba50e8606df3b3dfa27641bb62ff7420d045112c0c99c4d238df0ba" exitCode=0
Sep 14 13:17:33.697482 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.697161    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm" event={"ID":"3a02d0d5-539f-493b-8798-50054df800a3","Type":"ContainerDied","Data":"9fb3f669aba50e8606df3b3dfa27641bb62ff7420d045112c0c99c4d238df0ba"}
Sep 14 13:17:33.697482 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.697242    2557 scope.go:117] "RemoveContainer" containerID="da75bad4cc9599e9eff55b0458640a7e8eb47473d1d1353ba4ded238773205f0"
Sep 14 13:17:33.773541 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.773522    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:17:33.805041 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.804983    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-workload-certs\") pod \"3a02d0d5-539f-493b-8798-50054df800a3\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") "
Sep 14 13:17:33.805041 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.805019    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-credential-socket\") pod \"3a02d0d5-539f-493b-8798-50054df800a3\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") "
Sep 14 13:17:33.805212 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.805074    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a02d0d5-539f-493b-8798-50054df800a3-istio-token\") pod \"3a02d0d5-539f-493b-8798-50054df800a3\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") "
Sep 14 13:17:33.805212 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.805118    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a02d0d5-539f-493b-8798-50054df800a3-istiod-ca-cert\") pod \"3a02d0d5-539f-493b-8798-50054df800a3\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") "
Sep 14 13:17:33.805212 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.805140    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtxtx\" (UniqueName: \"kubernetes.io/projected/3a02d0d5-539f-493b-8798-50054df800a3-kube-api-access-gtxtx\") pod \"3a02d0d5-539f-493b-8798-50054df800a3\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") "
Sep 14 13:17:33.805345 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.805219    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3a02d0d5-539f-493b-8798-50054df800a3" (UID: "3a02d0d5-539f-493b-8798-50054df800a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:17:33.805345 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.805267    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a02d0d5-539f-493b-8798-50054df800a3-istio-podinfo\") pod \"3a02d0d5-539f-493b-8798-50054df800a3\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") "
Sep 14 13:17:33.805345 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.805323    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-istio-envoy\") pod \"3a02d0d5-539f-493b-8798-50054df800a3\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") "
Sep 14 13:17:33.805509 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.805347    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-istio-data\") pod \"3a02d0d5-539f-493b-8798-50054df800a3\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") "
Sep 14 13:17:33.805509 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.805451    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-workload-socket\") pod \"3a02d0d5-539f-493b-8798-50054df800a3\" (UID: \"3a02d0d5-539f-493b-8798-50054df800a3\") "
Sep 14 13:17:33.805509 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.805462    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3a02d0d5-539f-493b-8798-50054df800a3" (UID: "3a02d0d5-539f-493b-8798-50054df800a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:17:33.806369 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.806138    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a02d0d5-539f-493b-8798-50054df800a3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3a02d0d5-539f-493b-8798-50054df800a3" (UID: "3a02d0d5-539f-493b-8798-50054df800a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:17:33.806471 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.806425    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a02d0d5-539f-493b-8798-50054df800a3-istiod-ca-cert\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:17:33.806471 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.806448    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-workload-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:17:33.806471 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.806462    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-credential-socket\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:17:33.806721 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.806689    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3a02d0d5-539f-493b-8798-50054df800a3" (UID: "3a02d0d5-539f-493b-8798-50054df800a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:17:33.807016 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.806964    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-istio-data" (OuterVolumeSpecName: "istio-data") pod "3a02d0d5-539f-493b-8798-50054df800a3" (UID: "3a02d0d5-539f-493b-8798-50054df800a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:17:33.808630 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.808067    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a02d0d5-539f-493b-8798-50054df800a3-istio-token" (OuterVolumeSpecName: "istio-token") pod "3a02d0d5-539f-493b-8798-50054df800a3" (UID: "3a02d0d5-539f-493b-8798-50054df800a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:17:33.808630 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.808374    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a02d0d5-539f-493b-8798-50054df800a3-kube-api-access-gtxtx" (OuterVolumeSpecName: "kube-api-access-gtxtx") pod "3a02d0d5-539f-493b-8798-50054df800a3" (UID: "3a02d0d5-539f-493b-8798-50054df800a3"). InnerVolumeSpecName "kube-api-access-gtxtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:17:33.817027 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.817000    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3a02d0d5-539f-493b-8798-50054df800a3" (UID: "3a02d0d5-539f-493b-8798-50054df800a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:17:33.817154 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.817036    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a02d0d5-539f-493b-8798-50054df800a3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3a02d0d5-539f-493b-8798-50054df800a3" (UID: "3a02d0d5-539f-493b-8798-50054df800a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 13:17:33.906928 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.906907    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-workload-socket\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:17:33.906928 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.906928    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a02d0d5-539f-493b-8798-50054df800a3-istio-token\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:17:33.907053 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.906938    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtxtx\" (UniqueName: \"kubernetes.io/projected/3a02d0d5-539f-493b-8798-50054df800a3-kube-api-access-gtxtx\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:17:33.907053 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.906947    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a02d0d5-539f-493b-8798-50054df800a3-istio-podinfo\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:17:33.907053 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.906954    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-istio-envoy\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:17:33.907053 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:33.906962    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a02d0d5-539f-493b-8798-50054df800a3-istio-data\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:17:34.703480 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:34.703457    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"
Sep 14 13:17:34.703480 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:34.703463    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm" event={"ID":"3a02d0d5-539f-493b-8798-50054df800a3","Type":"ContainerDied","Data":"8654c14def55e4ec0c49903bb6d9f94ab14a0ef6ab1abc44dc150ceed4cf5f9d"}
Sep 14 13:17:34.703932 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:34.703511    2557 scope.go:117] "RemoveContainer" containerID="9fb3f669aba50e8606df3b3dfa27641bb62ff7420d045112c0c99c4d238df0ba"
Sep 14 13:17:34.723719 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:34.723693    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"]
Sep 14 13:17:34.727049 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:34.727028    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-758ecaaa-openshift-default-7558f79656-6bwmm"]
Sep 14 13:17:36.049142 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:36.049109    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a02d0d5-539f-493b-8798-50054df800a3" path="/var/lib/kubelet/pods/3a02d0d5-539f-493b-8798-50054df800a3/volumes"
Sep 14 13:17:38.231184 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:38.231145    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:38.231184 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:38.231186    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:38.244016 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:38.243992    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:38.248527 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:38.248480    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ace1c6-sg85n"]
Sep 14 13:17:38.249222 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:38.249197    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a02d0d5-539f-493b-8798-50054df800a3" containerName="istio-proxy"
Sep 14 13:17:38.249222 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:38.249220    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a02d0d5-539f-493b-8798-50054df800a3" containerName="istio-proxy"
Sep 14 13:17:38.249428 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:38.249234    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a02d0d5-539f-493b-8798-50054df800a3" containerName="istio-proxy"
Sep 14 13:17:38.249428 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:38.249244    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a02d0d5-539f-493b-8798-50054df800a3" containerName="istio-proxy"
Sep 14 13:17:38.249428 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:38.249330    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a02d0d5-539f-493b-8798-50054df800a3" containerName="istio-proxy"
Sep 14 13:17:38.253886 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:38.253867    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ace1c6-sg85n"
Sep 14 13:17:38.256677 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:38.256648    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbzp\""
Sep 14 13:17:38.259628 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:38.259562    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ace1c6-sg85n"]
Sep 14 13:17:38.344381 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:38.344348    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/150cd0fb-0fb9-499f-9865-e9707d79fa99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-ace1c6-sg85n\" (UID: \"150cd0fb-0fb9-499f-9865-e9707d79fa99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ace1c6-sg85n"
Sep 14 13:17:38.445967 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:38.445926    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/150cd0fb-0fb9-499f-9865-e9707d79fa99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-ace1c6-sg85n\" (UID: \"150cd0fb-0fb9-499f-9865-e9707d79fa99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ace1c6-sg85n"
Sep 14 13:17:38.446656 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:38.446636    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/150cd0fb-0fb9-499f-9865-e9707d79fa99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-ace1c6-sg85n\" (UID: \"150cd0fb-0fb9-499f-9865-e9707d79fa99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ace1c6-sg85n"
Sep 14 13:17:38.568479 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:38.568403    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ace1c6-sg85n"
Sep 14 13:17:38.737860 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:38.737828    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:17:38.841130 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:38.841108    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ace1c6-sg85n"]
Sep 14 13:17:38.843191 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:17:38.843163    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150cd0fb_0fb9_499f_9865_e9707d79fa99.slice/crio-66604a9780479675eb527f653e3c6b301b069785b7b21cc42c94b85e2dee2bc8 WatchSource:0}: Error finding container 66604a9780479675eb527f653e3c6b301b069785b7b21cc42c94b85e2dee2bc8: Status 404 returned error can't find the container with id 66604a9780479675eb527f653e3c6b301b069785b7b21cc42c94b85e2dee2bc8
Sep 14 13:17:39.734093 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:39.734058    2557 generic.go:356] "Generic (PLEG): container finished" podID="150cd0fb-0fb9-499f-9865-e9707d79fa99" containerID="f9bc123b34ec7bba30874180af9631edbaa265b940428a1a0ac2b7172d1d4084" exitCode=0
Sep 14 13:17:39.734420 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:39.734139    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ace1c6-sg85n" event={"ID":"150cd0fb-0fb9-499f-9865-e9707d79fa99","Type":"ContainerDied","Data":"f9bc123b34ec7bba30874180af9631edbaa265b940428a1a0ac2b7172d1d4084"}
Sep 14 13:17:39.734420 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:39.734196    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ace1c6-sg85n" event={"ID":"150cd0fb-0fb9-499f-9865-e9707d79fa99","Type":"ContainerStarted","Data":"66604a9780479675eb527f653e3c6b301b069785b7b21cc42c94b85e2dee2bc8"}
Sep 14 13:17:40.879234 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:40.879209    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ace1c6-sg85n"
Sep 14 13:17:40.968918 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:40.968893    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/150cd0fb-0fb9-499f-9865-e9707d79fa99-maas-api-service-ca\") pod \"150cd0fb-0fb9-499f-9865-e9707d79fa99\" (UID: \"150cd0fb-0fb9-499f-9865-e9707d79fa99\") "
Sep 14 13:17:40.969224 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:40.969204    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/150cd0fb-0fb9-499f-9865-e9707d79fa99-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "150cd0fb-0fb9-499f-9865-e9707d79fa99" (UID: "150cd0fb-0fb9-499f-9865-e9707d79fa99"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:17:41.070250 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:41.070187    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/150cd0fb-0fb9-499f-9865-e9707d79fa99-maas-api-service-ca\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:17:41.746545 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:41.746519    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ace1c6-sg85n"
Sep 14 13:17:41.746714 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:41.746556    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ace1c6-sg85n" event={"ID":"150cd0fb-0fb9-499f-9865-e9707d79fa99","Type":"ContainerDied","Data":"66604a9780479675eb527f653e3c6b301b069785b7b21cc42c94b85e2dee2bc8"}
Sep 14 13:17:41.746714 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:41.746598    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66604a9780479675eb527f653e3c6b301b069785b7b21cc42c94b85e2dee2bc8"
Sep 14 13:17:48.644534 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:48.644495    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl"]
Sep 14 13:17:48.645148 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:48.644764    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl" podUID="04671737-f924-4fc0-a8e6-90f62e4944cd" containerName="payload-pre-processing" containerID="cri-o://ae1af51c46798ddd9796d543cff81efd0e233da1c9bcfd43ca6c23ceba440d2c" gracePeriod=30
Sep 14 13:17:48.791418 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:48.791366    2557 generic.go:356] "Generic (PLEG): container finished" podID="04671737-f924-4fc0-a8e6-90f62e4944cd" containerID="ae1af51c46798ddd9796d543cff81efd0e233da1c9bcfd43ca6c23ceba440d2c" exitCode=0
Sep 14 13:17:48.791595 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:48.791443    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl" event={"ID":"04671737-f924-4fc0-a8e6-90f62e4944cd","Type":"ContainerDied","Data":"ae1af51c46798ddd9796d543cff81efd0e233da1c9bcfd43ca6c23ceba440d2c"}
Sep 14 13:17:48.814976 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:48.814948    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ace1c6-sg85n"]
Sep 14 13:17:48.818040 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:48.818010    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ace1c6-sg85n"]
Sep 14 13:17:48.916538 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:48.916513    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl"
Sep 14 13:17:49.042292 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:49.042262    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77b8n\" (UniqueName: \"kubernetes.io/projected/04671737-f924-4fc0-a8e6-90f62e4944cd-kube-api-access-77b8n\") pod \"04671737-f924-4fc0-a8e6-90f62e4944cd\" (UID: \"04671737-f924-4fc0-a8e6-90f62e4944cd\") "
Sep 14 13:17:49.042499 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:49.042315    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04671737-f924-4fc0-a8e6-90f62e4944cd-plugins-config-volume\") pod \"04671737-f924-4fc0-a8e6-90f62e4944cd\" (UID: \"04671737-f924-4fc0-a8e6-90f62e4944cd\") "
Sep 14 13:17:49.042649 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:49.042625    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04671737-f924-4fc0-a8e6-90f62e4944cd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "04671737-f924-4fc0-a8e6-90f62e4944cd" (UID: "04671737-f924-4fc0-a8e6-90f62e4944cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:17:49.044250 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:49.044227    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04671737-f924-4fc0-a8e6-90f62e4944cd-kube-api-access-77b8n" (OuterVolumeSpecName: "kube-api-access-77b8n") pod "04671737-f924-4fc0-a8e6-90f62e4944cd" (UID: "04671737-f924-4fc0-a8e6-90f62e4944cd"). InnerVolumeSpecName "kube-api-access-77b8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:17:49.143271 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:49.143223    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77b8n\" (UniqueName: \"kubernetes.io/projected/04671737-f924-4fc0-a8e6-90f62e4944cd-kube-api-access-77b8n\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:17:49.143271 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:49.143246    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04671737-f924-4fc0-a8e6-90f62e4944cd-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:17:49.798571 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:49.798544    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl"
Sep 14 13:17:49.799074 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:49.798632    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl" event={"ID":"04671737-f924-4fc0-a8e6-90f62e4944cd","Type":"ContainerDied","Data":"0ebafe3b5307a77572d2846122ecce284a09fe8d5a9599dec8be312e492195af"}
Sep 14 13:17:49.799074 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:49.798680    2557 scope.go:117] "RemoveContainer" containerID="ae1af51c46798ddd9796d543cff81efd0e233da1c9bcfd43ca6c23ceba440d2c"
Sep 14 13:17:49.821028 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:49.821004    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl"]
Sep 14 13:17:49.824007 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:49.823989    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ace1c6-df7b95966-ddphl"]
Sep 14 13:17:50.050380 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.050304    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04671737-f924-4fc0-a8e6-90f62e4944cd" path="/var/lib/kubelet/pods/04671737-f924-4fc0-a8e6-90f62e4944cd/volumes"
Sep 14 13:17:50.050643 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.050630    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="150cd0fb-0fb9-499f-9865-e9707d79fa99" path="/var/lib/kubelet/pods/150cd0fb-0fb9-499f-9865-e9707d79fa99/volumes"
Sep 14 13:17:50.264978 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.264947    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2"]
Sep 14 13:17:50.265353 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.265341    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="150cd0fb-0fb9-499f-9865-e9707d79fa99" containerName="revoke-keys"
Sep 14 13:17:50.265398 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.265355    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="150cd0fb-0fb9-499f-9865-e9707d79fa99" containerName="revoke-keys"
Sep 14 13:17:50.265398 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.265380    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04671737-f924-4fc0-a8e6-90f62e4944cd" containerName="payload-pre-processing"
Sep 14 13:17:50.265398 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.265385    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="04671737-f924-4fc0-a8e6-90f62e4944cd" containerName="payload-pre-processing"
Sep 14 13:17:50.265498 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.265450    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="04671737-f924-4fc0-a8e6-90f62e4944cd" containerName="payload-pre-processing"
Sep 14 13:17:50.265498 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.265463    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a02d0d5-539f-493b-8798-50054df800a3" containerName="istio-proxy"
Sep 14 13:17:50.265498 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.265468    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="150cd0fb-0fb9-499f-9865-e9707d79fa99" containerName="revoke-keys"
Sep 14 13:17:50.270042 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.270020    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2"
Sep 14 13:17:50.272863 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.272837    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-31063a\""
Sep 14 13:17:50.273000 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.272934    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-31063a-dockercfg-584qh\""
Sep 14 13:17:50.276715 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.276696    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2"]
Sep 14 13:17:50.299768 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.299729    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9"]
Sep 14 13:17:50.303170 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.303132    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9"
Sep 14 13:17:50.313429 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.313407    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9"]
Sep 14 13:17:50.355657 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.355625    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5h8x\" (UniqueName: \"kubernetes.io/projected/0d1a61ee-a98b-4d49-87e9-6700bce75a1c-kube-api-access-d5h8x\") pod \"payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9\" (UID: \"0d1a61ee-a98b-4d49-87e9-6700bce75a1c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9"
Sep 14 13:17:50.355843 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.355666    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1a61ee-a98b-4d49-87e9-6700bce75a1c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9\" (UID: \"0d1a61ee-a98b-4d49-87e9-6700bce75a1c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9"
Sep 14 13:17:50.355843 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.355713    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt9px\" (UniqueName: \"kubernetes.io/projected/d272c237-fddc-42d5-ad6f-c222c3370b96-kube-api-access-lt9px\") pod \"payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2\" (UID: \"d272c237-fddc-42d5-ad6f-c222c3370b96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2"
Sep 14 13:17:50.355843 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.355803    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d272c237-fddc-42d5-ad6f-c222c3370b96-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2\" (UID: \"d272c237-fddc-42d5-ad6f-c222c3370b96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2"
Sep 14 13:17:50.456559 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.456525    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5h8x\" (UniqueName: \"kubernetes.io/projected/0d1a61ee-a98b-4d49-87e9-6700bce75a1c-kube-api-access-d5h8x\") pod \"payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9\" (UID: \"0d1a61ee-a98b-4d49-87e9-6700bce75a1c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9"
Sep 14 13:17:50.456559 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.456567    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1a61ee-a98b-4d49-87e9-6700bce75a1c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9\" (UID: \"0d1a61ee-a98b-4d49-87e9-6700bce75a1c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9"
Sep 14 13:17:50.456849 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.456734    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt9px\" (UniqueName: \"kubernetes.io/projected/d272c237-fddc-42d5-ad6f-c222c3370b96-kube-api-access-lt9px\") pod \"payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2\" (UID: \"d272c237-fddc-42d5-ad6f-c222c3370b96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2"
Sep 14 13:17:50.456849 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.456805    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d272c237-fddc-42d5-ad6f-c222c3370b96-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2\" (UID: \"d272c237-fddc-42d5-ad6f-c222c3370b96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2"
Sep 14 13:17:50.457299 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.457282    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1a61ee-a98b-4d49-87e9-6700bce75a1c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9\" (UID: \"0d1a61ee-a98b-4d49-87e9-6700bce75a1c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9"
Sep 14 13:17:50.457729 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.457680    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d272c237-fddc-42d5-ad6f-c222c3370b96-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2\" (UID: \"d272c237-fddc-42d5-ad6f-c222c3370b96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2"
Sep 14 13:17:50.465851 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.465825    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt9px\" (UniqueName: \"kubernetes.io/projected/d272c237-fddc-42d5-ad6f-c222c3370b96-kube-api-access-lt9px\") pod \"payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2\" (UID: \"d272c237-fddc-42d5-ad6f-c222c3370b96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2"
Sep 14 13:17:50.466315 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.466292    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5h8x\" (UniqueName: \"kubernetes.io/projected/0d1a61ee-a98b-4d49-87e9-6700bce75a1c-kube-api-access-d5h8x\") pod \"payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9\" (UID: \"0d1a61ee-a98b-4d49-87e9-6700bce75a1c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9"
Sep 14 13:17:50.581405 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.581331    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2"
Sep 14 13:17:50.613774 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.613723    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9"
Sep 14 13:17:50.859968 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.859894    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2"]
Sep 14 13:17:50.862493 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:17:50.862464    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd272c237_fddc_42d5_ad6f_c222c3370b96.slice/crio-e7476c7d1ead3e806eeb2b445d28f9419240bd53135dc2449d6a2bb836bf0f82 WatchSource:0}: Error finding container e7476c7d1ead3e806eeb2b445d28f9419240bd53135dc2449d6a2bb836bf0f82: Status 404 returned error can't find the container with id e7476c7d1ead3e806eeb2b445d28f9419240bd53135dc2449d6a2bb836bf0f82
Sep 14 13:17:50.898717 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:50.898698    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9"]
Sep 14 13:17:50.900892 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:17:50.900870    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1a61ee_a98b_4d49_87e9_6700bce75a1c.slice/crio-a002f3ac51e6dc54e3102e2ef7447ad60d58451f1670fd87b8645a34335d4331 WatchSource:0}: Error finding container a002f3ac51e6dc54e3102e2ef7447ad60d58451f1670fd87b8645a34335d4331: Status 404 returned error can't find the container with id a002f3ac51e6dc54e3102e2ef7447ad60d58451f1670fd87b8645a34335d4331
Sep 14 13:17:51.812667 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:51.812630    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9" event={"ID":"0d1a61ee-a98b-4d49-87e9-6700bce75a1c","Type":"ContainerStarted","Data":"afea69578344fff65dfb420f615e89c17ef79f7d6b13330df2a476bce8af1870"}
Sep 14 13:17:51.812857 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:51.812691    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9" event={"ID":"0d1a61ee-a98b-4d49-87e9-6700bce75a1c","Type":"ContainerStarted","Data":"a002f3ac51e6dc54e3102e2ef7447ad60d58451f1670fd87b8645a34335d4331"}
Sep 14 13:17:51.812925 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:51.812853    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9"
Sep 14 13:17:51.814465 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:51.814441    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2" event={"ID":"d272c237-fddc-42d5-ad6f-c222c3370b96","Type":"ContainerStarted","Data":"2c4d3c53bf32efe1c18bd7a7facfb588a47aeb7fc512bc999cbbcbe447e73b51"}
Sep 14 13:17:51.814596 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:51.814488    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2" event={"ID":"d272c237-fddc-42d5-ad6f-c222c3370b96","Type":"ContainerStarted","Data":"e7476c7d1ead3e806eeb2b445d28f9419240bd53135dc2449d6a2bb836bf0f82"}
Sep 14 13:17:51.814596 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:51.814521    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2"
Sep 14 13:17:51.844317 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:51.844266    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2" podStartSLOduration=1.84425133 podStartE2EDuration="1.84425133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:17:51.843181532 +0000 UTC m=+1190.277857052" watchObservedRunningTime="2026-09-14 13:17:51.84425133 +0000 UTC m=+1190.278926885"
Sep 14 13:17:51.844658 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:17:51.844629    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9" podStartSLOduration=1.844619086 podStartE2EDuration="1.844619086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:17:51.828107004 +0000 UTC m=+1190.262782524" watchObservedRunningTime="2026-09-14 13:17:51.844619086 +0000 UTC m=+1190.279294601"
Sep 14 13:18:02.088654 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:02.088619    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwqnf_62d953d6-3b86-4c95-9a47-9579d88993a7/ovn-acl-logging/0.log"
Sep 14 13:18:02.093907 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:02.093887    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwqnf_62d953d6-3b86-4c95-9a47-9579d88993a7/ovn-acl-logging/0.log"
Sep 14 13:18:02.822149 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:02.822119    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9"
Sep 14 13:18:02.822322 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:02.822194    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2"
Sep 14 13:18:12.817188 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:12.817156    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-975227-wfttp"]
Sep 14 13:18:12.820880 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:12.820860    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-975227-wfttp"
Sep 14 13:18:12.823593 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:12.823570    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbzp\""
Sep 14 13:18:12.828424 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:12.828398    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-975227-wfttp"]
Sep 14 13:18:12.873346 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:12.873312    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99081dab-7591-40bc-b006-e7197a2be08c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-975227-wfttp\" (UID: \"99081dab-7591-40bc-b006-e7197a2be08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-975227-wfttp"
Sep 14 13:18:12.974433 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:12.974395    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99081dab-7591-40bc-b006-e7197a2be08c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-975227-wfttp\" (UID: \"99081dab-7591-40bc-b006-e7197a2be08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-975227-wfttp"
Sep 14 13:18:12.975111 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:12.975083    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99081dab-7591-40bc-b006-e7197a2be08c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-975227-wfttp\" (UID: \"99081dab-7591-40bc-b006-e7197a2be08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-975227-wfttp"
Sep 14 13:18:13.132831 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:13.132722    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-975227-wfttp"
Sep 14 13:18:13.407893 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:13.407867    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-975227-wfttp"]
Sep 14 13:18:13.413572 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:18:13.413529    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99081dab_7591_40bc_b006_e7197a2be08c.slice/crio-e89d0f553b63c0a7f115d27fc719a1539db19bf376904ec329e5c7bf9128395d WatchSource:0}: Error finding container e89d0f553b63c0a7f115d27fc719a1539db19bf376904ec329e5c7bf9128395d: Status 404 returned error can't find the container with id e89d0f553b63c0a7f115d27fc719a1539db19bf376904ec329e5c7bf9128395d
Sep 14 13:18:13.946695 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:13.946662    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-975227-wfttp" event={"ID":"99081dab-7591-40bc-b006-e7197a2be08c","Type":"ContainerStarted","Data":"f84091a8b1e1c5d4d817e6ed1c6b254bc24147a005621790a6c7e147d6ed611d"}
Sep 14 13:18:13.947188 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:13.946716    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-975227-wfttp" event={"ID":"99081dab-7591-40bc-b006-e7197a2be08c","Type":"ContainerStarted","Data":"e89d0f553b63c0a7f115d27fc719a1539db19bf376904ec329e5c7bf9128395d"}
Sep 14 13:18:13.966348 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:13.966303    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-975227-wfttp" podStartSLOduration=1.966287993 podStartE2EDuration="1.966287993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:18:13.964726917 +0000 UTC m=+1212.399402437" watchObservedRunningTime="2026-09-14 13:18:13.966287993 +0000 UTC m=+1212.400963512"
Sep 14 13:18:14.955107 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:14.955070    2557 generic.go:356] "Generic (PLEG): container finished" podID="99081dab-7591-40bc-b006-e7197a2be08c" containerID="f84091a8b1e1c5d4d817e6ed1c6b254bc24147a005621790a6c7e147d6ed611d" exitCode=0
Sep 14 13:18:14.955454 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:14.955134    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-975227-wfttp" event={"ID":"99081dab-7591-40bc-b006-e7197a2be08c","Type":"ContainerDied","Data":"f84091a8b1e1c5d4d817e6ed1c6b254bc24147a005621790a6c7e147d6ed611d"}
Sep 14 13:18:16.102787 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:16.102765    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-975227-wfttp"
Sep 14 13:18:16.204752 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:16.204709    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99081dab-7591-40bc-b006-e7197a2be08c-maas-api-service-ca\") pod \"99081dab-7591-40bc-b006-e7197a2be08c\" (UID: \"99081dab-7591-40bc-b006-e7197a2be08c\") "
Sep 14 13:18:16.205033 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:16.205012    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99081dab-7591-40bc-b006-e7197a2be08c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "99081dab-7591-40bc-b006-e7197a2be08c" (UID: "99081dab-7591-40bc-b006-e7197a2be08c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:18:16.306291 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:16.306227    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99081dab-7591-40bc-b006-e7197a2be08c-maas-api-service-ca\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:18:16.967861 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:16.967832    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-975227-wfttp"
Sep 14 13:18:16.967861 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:16.967848    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-975227-wfttp" event={"ID":"99081dab-7591-40bc-b006-e7197a2be08c","Type":"ContainerDied","Data":"e89d0f553b63c0a7f115d27fc719a1539db19bf376904ec329e5c7bf9128395d"}
Sep 14 13:18:16.968055 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:16.967889    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e89d0f553b63c0a7f115d27fc719a1539db19bf376904ec329e5c7bf9128395d"
Sep 14 13:18:25.220384 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:25.220353    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7"]
Sep 14 13:18:25.220879 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:25.220768    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99081dab-7591-40bc-b006-e7197a2be08c" containerName="revoke-keys"
Sep 14 13:18:25.220879 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:25.220781    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="99081dab-7591-40bc-b006-e7197a2be08c" containerName="revoke-keys"
Sep 14 13:18:25.220879 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:25.220859    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="99081dab-7591-40bc-b006-e7197a2be08c" containerName="revoke-keys"
Sep 14 13:18:25.224071 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:25.224054    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7"
Sep 14 13:18:25.227506 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:25.227484    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-b8be88-dockercfg-f75jp\""
Sep 14 13:18:25.227626 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:25.227579    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-b8be88\""
Sep 14 13:18:25.239637 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:25.239617    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7"]
Sep 14 13:18:25.289157 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:25.289124    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1137a84f-0a7a-4a08-b43f-5d6273537a10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7\" (UID: \"1137a84f-0a7a-4a08-b43f-5d6273537a10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7"
Sep 14 13:18:25.289323 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:25.289173    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flb9q\" (UniqueName: \"kubernetes.io/projected/1137a84f-0a7a-4a08-b43f-5d6273537a10-kube-api-access-flb9q\") pod \"payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7\" (UID: \"1137a84f-0a7a-4a08-b43f-5d6273537a10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7"
Sep 14 13:18:25.390324 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:25.390289    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flb9q\" (UniqueName: \"kubernetes.io/projected/1137a84f-0a7a-4a08-b43f-5d6273537a10-kube-api-access-flb9q\") pod \"payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7\" (UID: \"1137a84f-0a7a-4a08-b43f-5d6273537a10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7"
Sep 14 13:18:25.390489 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:25.390464    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1137a84f-0a7a-4a08-b43f-5d6273537a10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7\" (UID: \"1137a84f-0a7a-4a08-b43f-5d6273537a10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7"
Sep 14 13:18:25.391377 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:25.391355    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1137a84f-0a7a-4a08-b43f-5d6273537a10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7\" (UID: \"1137a84f-0a7a-4a08-b43f-5d6273537a10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7"
Sep 14 13:18:25.414152 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:25.414124    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flb9q\" (UniqueName: \"kubernetes.io/projected/1137a84f-0a7a-4a08-b43f-5d6273537a10-kube-api-access-flb9q\") pod \"payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7\" (UID: \"1137a84f-0a7a-4a08-b43f-5d6273537a10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7"
Sep 14 13:18:25.465504 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:25.465468    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-975227-29823195-rk8r7"]
Sep 14 13:18:25.467934 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:25.467866    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-975227-29823195-rk8r7"]
Sep 14 13:18:25.534110 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:25.534033    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7"
Sep 14 13:18:25.768128 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:25.768093    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-975227-wfttp"]
Sep 14 13:18:25.775533 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:25.775509    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-975227-wfttp"]
Sep 14 13:18:25.844811 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:25.844710    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7"]
Sep 14 13:18:26.021272 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:26.021237    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7" event={"ID":"1137a84f-0a7a-4a08-b43f-5d6273537a10","Type":"ContainerStarted","Data":"7fa8c3e4d0cdaa888e1bf2b63be1cd5b9400899c0d8a4854fc72cc7572570145"}
Sep 14 13:18:26.021437 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:26.021291    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7" event={"ID":"1137a84f-0a7a-4a08-b43f-5d6273537a10","Type":"ContainerStarted","Data":"af3bbe78625495e3d8254169d976533f435b31116d5bf2d53053639642f65236"}
Sep 14 13:18:26.021437 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:26.021376    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7"
Sep 14 13:18:26.051051 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:26.051006    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7" podStartSLOduration=1.050992518 podStartE2EDuration="1.050992518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:18:26.049423901 +0000 UTC m=+1224.484099419" watchObservedRunningTime="2026-09-14 13:18:26.050992518 +0000 UTC m=+1224.485668054"
Sep 14 13:18:26.051205 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:26.051028    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99081dab-7591-40bc-b006-e7197a2be08c" path="/var/lib/kubelet/pods/99081dab-7591-40bc-b006-e7197a2be08c/volumes"
Sep 14 13:18:26.051545 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:26.051529    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c27855c4-c530-4798-b25a-b8bda3cbea9f" path="/var/lib/kubelet/pods/c27855c4-c530-4798-b25a-b8bda3cbea9f/volumes"
Sep 14 13:18:33.611046 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:33.610999    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"]
Sep 14 13:18:33.611466 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:33.611260    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b" podUID="b70757db-81cb-40e5-9ebb-d98486fb7f97" containerName="istio-proxy" containerID="cri-o://02d177400785a1665929d1cf8d7c4008c5ee4773c0dd6f96b3afb62085da70a7" gracePeriod=30
Sep 14 13:18:33.942829 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:33.942796    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cadd9c-zrvjl"]
Sep 14 13:18:33.946028 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:33.946012    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cadd9c-zrvjl"
Sep 14 13:18:33.948983 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:33.948958    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbzp\""
Sep 14 13:18:33.957088 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:33.957063    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cadd9c-zrvjl"]
Sep 14 13:18:34.073162 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:34.073129    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e338e891-e807-4307-aa85-6ce7c7a59733-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cadd9c-zrvjl\" (UID: \"e338e891-e807-4307-aa85-6ce7c7a59733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cadd9c-zrvjl"
Sep 14 13:18:34.174662 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:34.174629    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e338e891-e807-4307-aa85-6ce7c7a59733-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cadd9c-zrvjl\" (UID: \"e338e891-e807-4307-aa85-6ce7c7a59733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cadd9c-zrvjl"
Sep 14 13:18:34.175286 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:34.175259    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e338e891-e807-4307-aa85-6ce7c7a59733-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cadd9c-zrvjl\" (UID: \"e338e891-e807-4307-aa85-6ce7c7a59733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cadd9c-zrvjl"
Sep 14 13:18:34.259771 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:34.259664    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cadd9c-zrvjl"
Sep 14 13:18:34.550684 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:34.550657    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cadd9c-zrvjl"]
Sep 14 13:18:34.552508 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:18:34.552482    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode338e891_e807_4307_aa85_6ce7c7a59733.slice/crio-c27e46c371b0411d5fabf6340cb6b498fc578839ee30e12067550eadc9b4c83d WatchSource:0}: Error finding container c27e46c371b0411d5fabf6340cb6b498fc578839ee30e12067550eadc9b4c83d: Status 404 returned error can't find the container with id c27e46c371b0411d5fabf6340cb6b498fc578839ee30e12067550eadc9b4c83d
Sep 14 13:18:35.086479 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:35.086436    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cadd9c-zrvjl" event={"ID":"e338e891-e807-4307-aa85-6ce7c7a59733","Type":"ContainerStarted","Data":"4095ec93a0e0cc0f0744279a4c53f24ad8ed1efbd46df7c9f918eba731503e88"}
Sep 14 13:18:35.086906 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:35.086507    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cadd9c-zrvjl" event={"ID":"e338e891-e807-4307-aa85-6ce7c7a59733","Type":"ContainerStarted","Data":"c27e46c371b0411d5fabf6340cb6b498fc578839ee30e12067550eadc9b4c83d"}
Sep 14 13:18:35.111524 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:35.111473    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cadd9c-zrvjl" podStartSLOduration=2.111457094 podStartE2EDuration="2.111457094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:18:35.109269732 +0000 UTC m=+1233.543945251" watchObservedRunningTime="2026-09-14 13:18:35.111457094 +0000 UTC m=+1233.546132612"
Sep 14 13:18:35.619576 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:35.619542    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"]
Sep 14 13:18:36.093683 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:36.093647    2557 generic.go:356] "Generic (PLEG): container finished" podID="e338e891-e807-4307-aa85-6ce7c7a59733" containerID="4095ec93a0e0cc0f0744279a4c53f24ad8ed1efbd46df7c9f918eba731503e88" exitCode=0
Sep 14 13:18:36.094110 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:36.093724    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cadd9c-zrvjl" event={"ID":"e338e891-e807-4307-aa85-6ce7c7a59733","Type":"ContainerDied","Data":"4095ec93a0e0cc0f0744279a4c53f24ad8ed1efbd46df7c9f918eba731503e88"}
Sep 14 13:18:37.029507 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:37.029474    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7"
Sep 14 13:18:37.244319 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:37.244288    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cadd9c-zrvjl"
Sep 14 13:18:37.406651 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:37.406568    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e338e891-e807-4307-aa85-6ce7c7a59733-maas-api-service-ca\") pod \"e338e891-e807-4307-aa85-6ce7c7a59733\" (UID: \"e338e891-e807-4307-aa85-6ce7c7a59733\") "
Sep 14 13:18:37.406982 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:37.406960    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e338e891-e807-4307-aa85-6ce7c7a59733-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e338e891-e807-4307-aa85-6ce7c7a59733" (UID: "e338e891-e807-4307-aa85-6ce7c7a59733"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:18:37.508151 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:37.508116    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e338e891-e807-4307-aa85-6ce7c7a59733-maas-api-service-ca\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:18:38.108492 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:38.108466    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cadd9c-zrvjl"
Sep 14 13:18:38.108646 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:38.108496    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cadd9c-zrvjl" event={"ID":"e338e891-e807-4307-aa85-6ce7c7a59733","Type":"ContainerDied","Data":"c27e46c371b0411d5fabf6340cb6b498fc578839ee30e12067550eadc9b4c83d"}
Sep 14 13:18:38.108646 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:38.108542    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c27e46c371b0411d5fabf6340cb6b498fc578839ee30e12067550eadc9b4c83d"
Sep 14 13:18:38.864646 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:38.864620    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:18:39.023546 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.023515    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfqjv\" (UniqueName: \"kubernetes.io/projected/b70757db-81cb-40e5-9ebb-d98486fb7f97-kube-api-access-lfqjv\") pod \"b70757db-81cb-40e5-9ebb-d98486fb7f97\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") "
Sep 14 13:18:39.023701 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.023570    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-workload-certs\") pod \"b70757db-81cb-40e5-9ebb-d98486fb7f97\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") "
Sep 14 13:18:39.023701 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.023600    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-envoy\") pod \"b70757db-81cb-40e5-9ebb-d98486fb7f97\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") "
Sep 14 13:18:39.023826 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.023719    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-data\") pod \"b70757db-81cb-40e5-9ebb-d98486fb7f97\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") "
Sep 14 13:18:39.023826 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.023808    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b70757db-81cb-40e5-9ebb-d98486fb7f97-istiod-ca-cert\") pod \"b70757db-81cb-40e5-9ebb-d98486fb7f97\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") "
Sep 14 13:18:39.023913 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.023868    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-workload-socket\") pod \"b70757db-81cb-40e5-9ebb-d98486fb7f97\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") "
Sep 14 13:18:39.023913 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.023897    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-podinfo\") pod \"b70757db-81cb-40e5-9ebb-d98486fb7f97\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") "
Sep 14 13:18:39.024022 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.023920    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b70757db-81cb-40e5-9ebb-d98486fb7f97" (UID: "b70757db-81cb-40e5-9ebb-d98486fb7f97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:18:39.024022 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.023948    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-data" (OuterVolumeSpecName: "istio-data") pod "b70757db-81cb-40e5-9ebb-d98486fb7f97" (UID: "b70757db-81cb-40e5-9ebb-d98486fb7f97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:18:39.024022 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.023995    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-credential-socket\") pod \"b70757db-81cb-40e5-9ebb-d98486fb7f97\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") "
Sep 14 13:18:39.024185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.024033    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-token\") pod \"b70757db-81cb-40e5-9ebb-d98486fb7f97\" (UID: \"b70757db-81cb-40e5-9ebb-d98486fb7f97\") "
Sep 14 13:18:39.024185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.024171    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b70757db-81cb-40e5-9ebb-d98486fb7f97" (UID: "b70757db-81cb-40e5-9ebb-d98486fb7f97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:18:39.024289 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.024184    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b70757db-81cb-40e5-9ebb-d98486fb7f97-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b70757db-81cb-40e5-9ebb-d98486fb7f97" (UID: "b70757db-81cb-40e5-9ebb-d98486fb7f97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:18:39.024289 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.024208    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b70757db-81cb-40e5-9ebb-d98486fb7f97" (UID: "b70757db-81cb-40e5-9ebb-d98486fb7f97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:18:39.024419 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.024404    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-workload-socket\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:18:39.024476 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.024421    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-credential-socket\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:18:39.024476 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.024433    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-workload-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:18:39.024476 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.024444    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-data\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:18:39.024476 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.024451    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b70757db-81cb-40e5-9ebb-d98486fb7f97-istiod-ca-cert\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:18:39.026189 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.026161    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b70757db-81cb-40e5-9ebb-d98486fb7f97" (UID: "b70757db-81cb-40e5-9ebb-d98486fb7f97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:18:39.026337 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.026321    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b70757db-81cb-40e5-9ebb-d98486fb7f97-kube-api-access-lfqjv" (OuterVolumeSpecName: "kube-api-access-lfqjv") pod "b70757db-81cb-40e5-9ebb-d98486fb7f97" (UID: "b70757db-81cb-40e5-9ebb-d98486fb7f97"). InnerVolumeSpecName "kube-api-access-lfqjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:18:39.026431 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.026416    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-token" (OuterVolumeSpecName: "istio-token") pod "b70757db-81cb-40e5-9ebb-d98486fb7f97" (UID: "b70757db-81cb-40e5-9ebb-d98486fb7f97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:18:39.026472 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.026437    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b70757db-81cb-40e5-9ebb-d98486fb7f97" (UID: "b70757db-81cb-40e5-9ebb-d98486fb7f97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 13:18:39.118851 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.116094    2557 generic.go:356] "Generic (PLEG): container finished" podID="b70757db-81cb-40e5-9ebb-d98486fb7f97" containerID="02d177400785a1665929d1cf8d7c4008c5ee4773c0dd6f96b3afb62085da70a7" exitCode=0
Sep 14 13:18:39.118851 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.116170    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b" event={"ID":"b70757db-81cb-40e5-9ebb-d98486fb7f97","Type":"ContainerDied","Data":"02d177400785a1665929d1cf8d7c4008c5ee4773c0dd6f96b3afb62085da70a7"}
Sep 14 13:18:39.118851 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.116221    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b" event={"ID":"b70757db-81cb-40e5-9ebb-d98486fb7f97","Type":"ContainerDied","Data":"6483f9b2b9e3b1d7ec412b0f12350cfcddf877ac85bd1b74a20871850863fd43"}
Sep 14 13:18:39.118851 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.116249    2557 scope.go:117] "RemoveContainer" containerID="02d177400785a1665929d1cf8d7c4008c5ee4773c0dd6f96b3afb62085da70a7"
Sep 14 13:18:39.118851 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.116396    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"
Sep 14 13:18:39.125618 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.125595    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-podinfo\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:18:39.125706 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.125624    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-token\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:18:39.125706 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.125638    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfqjv\" (UniqueName: \"kubernetes.io/projected/b70757db-81cb-40e5-9ebb-d98486fb7f97-kube-api-access-lfqjv\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:18:39.125706 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.125652    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b70757db-81cb-40e5-9ebb-d98486fb7f97-istio-envoy\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:18:39.129269 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.129251    2557 scope.go:117] "RemoveContainer" containerID="c98f27bf340ae742e0ae314a9369ebc3f1d9c92f03da3b4353c5fc86c7c99a68"
Sep 14 13:18:39.137184 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.137169    2557 scope.go:117] "RemoveContainer" containerID="02d177400785a1665929d1cf8d7c4008c5ee4773c0dd6f96b3afb62085da70a7"
Sep 14 13:18:39.137435 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:18:39.137419    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d177400785a1665929d1cf8d7c4008c5ee4773c0dd6f96b3afb62085da70a7\": container with ID starting with 02d177400785a1665929d1cf8d7c4008c5ee4773c0dd6f96b3afb62085da70a7 not found: ID does not exist" containerID="02d177400785a1665929d1cf8d7c4008c5ee4773c0dd6f96b3afb62085da70a7"
Sep 14 13:18:39.137480 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.137442    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d177400785a1665929d1cf8d7c4008c5ee4773c0dd6f96b3afb62085da70a7"} err="failed to get container status \"02d177400785a1665929d1cf8d7c4008c5ee4773c0dd6f96b3afb62085da70a7\": rpc error: code = NotFound desc = could not find container \"02d177400785a1665929d1cf8d7c4008c5ee4773c0dd6f96b3afb62085da70a7\": container with ID starting with 02d177400785a1665929d1cf8d7c4008c5ee4773c0dd6f96b3afb62085da70a7 not found: ID does not exist"
Sep 14 13:18:39.137480 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.137456    2557 scope.go:117] "RemoveContainer" containerID="c98f27bf340ae742e0ae314a9369ebc3f1d9c92f03da3b4353c5fc86c7c99a68"
Sep 14 13:18:39.137706 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:18:39.137689    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c98f27bf340ae742e0ae314a9369ebc3f1d9c92f03da3b4353c5fc86c7c99a68\": container with ID starting with c98f27bf340ae742e0ae314a9369ebc3f1d9c92f03da3b4353c5fc86c7c99a68 not found: ID does not exist" containerID="c98f27bf340ae742e0ae314a9369ebc3f1d9c92f03da3b4353c5fc86c7c99a68"
Sep 14 13:18:39.137842 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.137708    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98f27bf340ae742e0ae314a9369ebc3f1d9c92f03da3b4353c5fc86c7c99a68"} err="failed to get container status \"c98f27bf340ae742e0ae314a9369ebc3f1d9c92f03da3b4353c5fc86c7c99a68\": rpc error: code = NotFound desc = could not find container \"c98f27bf340ae742e0ae314a9369ebc3f1d9c92f03da3b4353c5fc86c7c99a68\": container with ID starting with c98f27bf340ae742e0ae314a9369ebc3f1d9c92f03da3b4353c5fc86c7c99a68 not found: ID does not exist"
Sep 14 13:18:39.144431 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.144410    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"]
Sep 14 13:18:39.150685 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:39.150663    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-975227-openshift-default-77867b86c4-bls4b"]
Sep 14 13:18:40.049873 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:40.049844    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b70757db-81cb-40e5-9ebb-d98486fb7f97" path="/var/lib/kubelet/pods/b70757db-81cb-40e5-9ebb-d98486fb7f97/volumes"
Sep 14 13:18:44.185845 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:44.185817    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cadd9c-29823195-2w9qz"]
Sep 14 13:18:44.189363 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:44.189332    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cadd9c-29823195-2w9qz"]
Sep 14 13:18:44.535556 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:44.535527    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cadd9c-zrvjl"]
Sep 14 13:18:44.540869 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:44.540845    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cadd9c-zrvjl"]
Sep 14 13:18:46.050546 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:46.050510    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e887453-6288-4733-9798-5d9fcfca0f0d" path="/var/lib/kubelet/pods/7e887453-6288-4733-9798-5d9fcfca0f0d/volumes"
Sep 14 13:18:46.050975 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:46.050956    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e338e891-e807-4307-aa85-6ce7c7a59733" path="/var/lib/kubelet/pods/e338e891-e807-4307-aa85-6ce7c7a59733/volumes"
Sep 14 13:18:50.638413 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:50.638377    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq" podUID="4af67e71-b311-45b8-8050-78c51f24ad1c" containerName="main" containerID="cri-o://27b9c9187fc9e69425c0ffc42a526dd5e98e3bdbdf314160546ca4fcf4f4ba82" gracePeriod=45
Sep 14 13:18:50.892268 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:50.892215    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:18:50.934374 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:50.934341    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-home\") pod \"4af67e71-b311-45b8-8050-78c51f24ad1c\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") "
Sep 14 13:18:50.934374 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:50.934379    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t52q4\" (UniqueName: \"kubernetes.io/projected/4af67e71-b311-45b8-8050-78c51f24ad1c-kube-api-access-t52q4\") pod \"4af67e71-b311-45b8-8050-78c51f24ad1c\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") "
Sep 14 13:18:50.934615 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:50.934399    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4af67e71-b311-45b8-8050-78c51f24ad1c-tls-certs\") pod \"4af67e71-b311-45b8-8050-78c51f24ad1c\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") "
Sep 14 13:18:50.934615 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:50.934480    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-dshm\") pod \"4af67e71-b311-45b8-8050-78c51f24ad1c\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") "
Sep 14 13:18:50.934615 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:50.934514    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-tmp-dir\") pod \"4af67e71-b311-45b8-8050-78c51f24ad1c\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") "
Sep 14 13:18:50.934615 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:50.934551    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-model-cache\") pod \"4af67e71-b311-45b8-8050-78c51f24ad1c\" (UID: \"4af67e71-b311-45b8-8050-78c51f24ad1c\") "
Sep 14 13:18:50.934615 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:50.934604    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-home" (OuterVolumeSpecName: "home") pod "4af67e71-b311-45b8-8050-78c51f24ad1c" (UID: "4af67e71-b311-45b8-8050-78c51f24ad1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:18:50.934924 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:50.934832    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-model-cache" (OuterVolumeSpecName: "model-cache") pod "4af67e71-b311-45b8-8050-78c51f24ad1c" (UID: "4af67e71-b311-45b8-8050-78c51f24ad1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:18:50.934924 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:50.934852    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4af67e71-b311-45b8-8050-78c51f24ad1c" (UID: "4af67e71-b311-45b8-8050-78c51f24ad1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:18:50.934924 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:50.934908    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-home\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:18:50.936604 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:50.936581    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-dshm" (OuterVolumeSpecName: "dshm") pod "4af67e71-b311-45b8-8050-78c51f24ad1c" (UID: "4af67e71-b311-45b8-8050-78c51f24ad1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:18:50.936604 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:50.936590    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4af67e71-b311-45b8-8050-78c51f24ad1c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4af67e71-b311-45b8-8050-78c51f24ad1c" (UID: "4af67e71-b311-45b8-8050-78c51f24ad1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:18:50.936819 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:50.936681    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af67e71-b311-45b8-8050-78c51f24ad1c-kube-api-access-t52q4" (OuterVolumeSpecName: "kube-api-access-t52q4") pod "4af67e71-b311-45b8-8050-78c51f24ad1c" (UID: "4af67e71-b311-45b8-8050-78c51f24ad1c"). InnerVolumeSpecName "kube-api-access-t52q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:18:51.035407 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:51.035384    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-dshm\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:18:51.035407 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:51.035406    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-tmp-dir\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:18:51.035543 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:51.035414    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4af67e71-b311-45b8-8050-78c51f24ad1c-model-cache\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:18:51.035543 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:51.035424    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t52q4\" (UniqueName: \"kubernetes.io/projected/4af67e71-b311-45b8-8050-78c51f24ad1c-kube-api-access-t52q4\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:18:51.035543 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:51.035432    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4af67e71-b311-45b8-8050-78c51f24ad1c-tls-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:18:51.187495 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:51.187432    2557 generic.go:356] "Generic (PLEG): container finished" podID="4af67e71-b311-45b8-8050-78c51f24ad1c" containerID="27b9c9187fc9e69425c0ffc42a526dd5e98e3bdbdf314160546ca4fcf4f4ba82" exitCode=0
Sep 14 13:18:51.187495 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:51.187475    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"
Sep 14 13:18:51.187686 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:51.187518    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq" event={"ID":"4af67e71-b311-45b8-8050-78c51f24ad1c","Type":"ContainerDied","Data":"27b9c9187fc9e69425c0ffc42a526dd5e98e3bdbdf314160546ca4fcf4f4ba82"}
Sep 14 13:18:51.187686 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:51.187568    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq" event={"ID":"4af67e71-b311-45b8-8050-78c51f24ad1c","Type":"ContainerDied","Data":"2cab59a35f9f87ff275ea56a2dd658e70aca6339bf15fc8b81064c6b872ec900"}
Sep 14 13:18:51.187686 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:51.187590    2557 scope.go:117] "RemoveContainer" containerID="27b9c9187fc9e69425c0ffc42a526dd5e98e3bdbdf314160546ca4fcf4f4ba82"
Sep 14 13:18:51.197049 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:51.197031    2557 scope.go:117] "RemoveContainer" containerID="27b9c9187fc9e69425c0ffc42a526dd5e98e3bdbdf314160546ca4fcf4f4ba82"
Sep 14 13:18:51.197306 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:18:51.197289    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b9c9187fc9e69425c0ffc42a526dd5e98e3bdbdf314160546ca4fcf4f4ba82\": container with ID starting with 27b9c9187fc9e69425c0ffc42a526dd5e98e3bdbdf314160546ca4fcf4f4ba82 not found: ID does not exist" containerID="27b9c9187fc9e69425c0ffc42a526dd5e98e3bdbdf314160546ca4fcf4f4ba82"
Sep 14 13:18:51.197355 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:51.197315    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b9c9187fc9e69425c0ffc42a526dd5e98e3bdbdf314160546ca4fcf4f4ba82"} err="failed to get container status \"27b9c9187fc9e69425c0ffc42a526dd5e98e3bdbdf314160546ca4fcf4f4ba82\": rpc error: code = NotFound desc = could not find container \"27b9c9187fc9e69425c0ffc42a526dd5e98e3bdbdf314160546ca4fcf4f4ba82\": container with ID starting with 27b9c9187fc9e69425c0ffc42a526dd5e98e3bdbdf314160546ca4fcf4f4ba82 not found: ID does not exist"
Sep 14 13:18:51.212835 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:51.212815    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"]
Sep 14 13:18:51.216668 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:51.216649    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-cadd9c/test-model-cadd9c-kserve-675546dfc7-kgqsq"]
Sep 14 13:18:52.050661 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:52.050624    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4af67e71-b311-45b8-8050-78c51f24ad1c" path="/var/lib/kubelet/pods/4af67e71-b311-45b8-8050-78c51f24ad1c/volumes"
Sep 14 13:18:58.966595 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:58.966523    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"]
Sep 14 13:18:58.967076 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:58.966940    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b70757db-81cb-40e5-9ebb-d98486fb7f97" containerName="istio-proxy"
Sep 14 13:18:58.967076 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:58.966953    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b70757db-81cb-40e5-9ebb-d98486fb7f97" containerName="istio-proxy"
Sep 14 13:18:58.967076 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:58.966963    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e338e891-e807-4307-aa85-6ce7c7a59733" containerName="revoke-keys"
Sep 14 13:18:58.967076 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:58.966969    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e338e891-e807-4307-aa85-6ce7c7a59733" containerName="revoke-keys"
Sep 14 13:18:58.967076 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:58.966991    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4af67e71-b311-45b8-8050-78c51f24ad1c" containerName="main"
Sep 14 13:18:58.967076 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:58.966997    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4af67e71-b311-45b8-8050-78c51f24ad1c" containerName="main"
Sep 14 13:18:58.967076 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:58.967054    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b70757db-81cb-40e5-9ebb-d98486fb7f97" containerName="istio-proxy"
Sep 14 13:18:58.967076 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:58.967063    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e338e891-e807-4307-aa85-6ce7c7a59733" containerName="revoke-keys"
Sep 14 13:18:58.967076 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:58.967071    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4af67e71-b311-45b8-8050-78c51f24ad1c" containerName="main"
Sep 14 13:18:58.971894 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:58.971860    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:58.974671 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:58.974639    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-14f3af-openshift-default-dockercfg-znm6s\""
Sep 14 13:18:58.985686 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:58.985658    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"]
Sep 14 13:18:59.007600 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.007576    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2336252-c624-48ac-8aef-599b3ebabd52-istio-token\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.007763 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.007658    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-credential-socket\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.007763 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.007688    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-istio-data\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.007763 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.007712    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2336252-c624-48ac-8aef-599b3ebabd52-istio-podinfo\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.007894 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.007768    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-istio-envoy\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.007894 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.007791    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktppn\" (UniqueName: \"kubernetes.io/projected/b2336252-c624-48ac-8aef-599b3ebabd52-kube-api-access-ktppn\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.007894 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.007814    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2336252-c624-48ac-8aef-599b3ebabd52-istiod-ca-cert\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.007894 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.007871    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-workload-socket\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.008049 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.007955    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-workload-certs\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.108448 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.108416    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-workload-certs\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.108632 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.108457    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2336252-c624-48ac-8aef-599b3ebabd52-istio-token\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.108632 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.108543    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-credential-socket\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.108632 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.108585    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-istio-data\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.108632 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.108614    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2336252-c624-48ac-8aef-599b3ebabd52-istio-podinfo\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.108858 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.108639    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-istio-envoy\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.108858 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.108772    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktppn\" (UniqueName: \"kubernetes.io/projected/b2336252-c624-48ac-8aef-599b3ebabd52-kube-api-access-ktppn\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.108858 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.108825    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2336252-c624-48ac-8aef-599b3ebabd52-istiod-ca-cert\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.108858 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.108854    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-workload-socket\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.109073 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.108891    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-workload-certs\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.109073 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.108945    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-credential-socket\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.109190 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.109147    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-istio-data\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.109270 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.109246    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-workload-socket\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.109506 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.109488    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2336252-c624-48ac-8aef-599b3ebabd52-istiod-ca-cert\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.110880 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.110861    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-istio-envoy\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.111095 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.111077    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2336252-c624-48ac-8aef-599b3ebabd52-istio-podinfo\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.117280 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.117259    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2336252-c624-48ac-8aef-599b3ebabd52-istio-token\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.117847 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.117826    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktppn\" (UniqueName: \"kubernetes.io/projected/b2336252-c624-48ac-8aef-599b3ebabd52-kube-api-access-ktppn\") pod \"e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") " pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.283164 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.283097    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:18:59.561316 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.561290    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"]
Sep 14 13:18:59.562573 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:18:59.562548    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2336252_c624_48ac_8aef_599b3ebabd52.slice/crio-c3ac782e8d46d851ff435af0c369e5c7f784e7da91733fa6aff1d55e59f5a760 WatchSource:0}: Error finding container c3ac782e8d46d851ff435af0c369e5c7f784e7da91733fa6aff1d55e59f5a760: Status 404 returned error can't find the container with id c3ac782e8d46d851ff435af0c369e5c7f784e7da91733fa6aff1d55e59f5a760
Sep 14 13:18:59.565189 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.565100    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:18:59.565280 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.565236    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:18:59.565322 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:18:59.565278    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:19:00.238816 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:00.238783    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j" event={"ID":"b2336252-c624-48ac-8aef-599b3ebabd52","Type":"ContainerStarted","Data":"e8927b6d25dcfe302e875df825a61aabe869a1fa0c7c9d97a9be8f55ff5e585d"}
Sep 14 13:19:00.239194 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:00.238851    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j" event={"ID":"b2336252-c624-48ac-8aef-599b3ebabd52","Type":"ContainerStarted","Data":"c3ac782e8d46d851ff435af0c369e5c7f784e7da91733fa6aff1d55e59f5a760"}
Sep 14 13:19:00.262811 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:00.262757    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j" podStartSLOduration=2.262729617 podStartE2EDuration="2.262729617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:19:00.261352496 +0000 UTC m=+1258.696028017" watchObservedRunningTime="2026-09-14 13:19:00.262729617 +0000 UTC m=+1258.697405136"
Sep 14 13:19:00.283570 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:00.283546    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:19:00.288712 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:00.288693    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:19:01.244197 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:01.244169    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:19:01.245143 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:01.245119    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:19:02.226022 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:02.225992    2557 scope.go:117] "RemoveContainer" containerID="c43b438a5cda0d7c276d1245762f6d6177b1e3e91fe9e664ba90e6f9115b2e1d"
Sep 14 13:19:07.312035 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:07.311992    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw"]
Sep 14 13:19:07.312664 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:07.312640    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b70757db-81cb-40e5-9ebb-d98486fb7f97" containerName="istio-proxy"
Sep 14 13:19:07.312664 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:07.312662    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b70757db-81cb-40e5-9ebb-d98486fb7f97" containerName="istio-proxy"
Sep 14 13:19:07.312946 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:07.312791    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b70757db-81cb-40e5-9ebb-d98486fb7f97" containerName="istio-proxy"
Sep 14 13:19:07.317915 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:07.317896    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw"
Sep 14 13:19:07.321529 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:07.321508    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-14f3af\""
Sep 14 13:19:07.321621 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:07.321553    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-14f3af-dockercfg-5tndg\""
Sep 14 13:19:07.332976 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:07.332951    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw"]
Sep 14 13:19:07.385398 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:07.385375    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28f1ea7b-a882-4dac-8e24-82f85023c5f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw\" (UID: \"28f1ea7b-a882-4dac-8e24-82f85023c5f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw"
Sep 14 13:19:07.385530 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:07.385454    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x99ld\" (UniqueName: \"kubernetes.io/projected/28f1ea7b-a882-4dac-8e24-82f85023c5f9-kube-api-access-x99ld\") pod \"payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw\" (UID: \"28f1ea7b-a882-4dac-8e24-82f85023c5f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw"
Sep 14 13:19:07.486010 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:07.485987    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28f1ea7b-a882-4dac-8e24-82f85023c5f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw\" (UID: \"28f1ea7b-a882-4dac-8e24-82f85023c5f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw"
Sep 14 13:19:07.486154 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:07.486043    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x99ld\" (UniqueName: \"kubernetes.io/projected/28f1ea7b-a882-4dac-8e24-82f85023c5f9-kube-api-access-x99ld\") pod \"payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw\" (UID: \"28f1ea7b-a882-4dac-8e24-82f85023c5f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw"
Sep 14 13:19:07.486693 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:07.486672    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28f1ea7b-a882-4dac-8e24-82f85023c5f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw\" (UID: \"28f1ea7b-a882-4dac-8e24-82f85023c5f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw"
Sep 14 13:19:07.499641 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:07.499621    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x99ld\" (UniqueName: \"kubernetes.io/projected/28f1ea7b-a882-4dac-8e24-82f85023c5f9-kube-api-access-x99ld\") pod \"payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw\" (UID: \"28f1ea7b-a882-4dac-8e24-82f85023c5f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw"
Sep 14 13:19:07.628888 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:07.628833    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw"
Sep 14 13:19:07.927732 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:07.927700    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw"]
Sep 14 13:19:07.928955 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:19:07.928914    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f1ea7b_a882_4dac_8e24_82f85023c5f9.slice/crio-aad11109733ecd3014b2bd62ec922c8b0ba5f475b38ca4f457e31b040ada77ff WatchSource:0}: Error finding container aad11109733ecd3014b2bd62ec922c8b0ba5f475b38ca4f457e31b040ada77ff: Status 404 returned error can't find the container with id aad11109733ecd3014b2bd62ec922c8b0ba5f475b38ca4f457e31b040ada77ff
Sep 14 13:19:08.291527 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:08.290188    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw" event={"ID":"28f1ea7b-a882-4dac-8e24-82f85023c5f9","Type":"ContainerStarted","Data":"0a3ac0e86faef4ae7772c4bcc77fef7586400ae7ede71925110f982228927b59"}
Sep 14 13:19:08.291527 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:08.290253    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw" event={"ID":"28f1ea7b-a882-4dac-8e24-82f85023c5f9","Type":"ContainerStarted","Data":"aad11109733ecd3014b2bd62ec922c8b0ba5f475b38ca4f457e31b040ada77ff"}
Sep 14 13:19:08.291527 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:08.291087    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw"
Sep 14 13:19:20.305141 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:20.305107    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw"
Sep 14 13:19:20.322405 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:20.322354    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw" podStartSLOduration=13.322340416 podStartE2EDuration="13.322340416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:19:08.321958795 +0000 UTC m=+1266.756634316" watchObservedRunningTime="2026-09-14 13:19:20.322340416 +0000 UTC m=+1278.757015935"
Sep 14 13:19:32.285902 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:32.285855    2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 14 13:19:32.286421 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:32.285930    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j" podUID="b2336252-c624-48ac-8aef-599b3ebabd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 14 13:19:34.444596 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:34.444556    2557 generic.go:356] "Generic (PLEG): container finished" podID="b2336252-c624-48ac-8aef-599b3ebabd52" containerID="e8927b6d25dcfe302e875df825a61aabe869a1fa0c7c9d97a9be8f55ff5e585d" exitCode=0
Sep 14 13:19:34.445060 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:34.444638    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j" event={"ID":"b2336252-c624-48ac-8aef-599b3ebabd52","Type":"ContainerDied","Data":"e8927b6d25dcfe302e875df825a61aabe869a1fa0c7c9d97a9be8f55ff5e585d"}
Sep 14 13:19:34.445154 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:34.445138    2557 scope.go:117] "RemoveContainer" containerID="e8927b6d25dcfe302e875df825a61aabe869a1fa0c7c9d97a9be8f55ff5e585d"
Sep 14 13:19:34.446310 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:34.446282    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:19:34.446359 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:34.446351    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:19:34.446405 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:34.446383    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:19:35.452917 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:35.452878    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j" event={"ID":"b2336252-c624-48ac-8aef-599b3ebabd52","Type":"ContainerStarted","Data":"ea840559eb85c589dad07863a0557b402c7b8efa90ca8002903be37b0e38257a"}
Sep 14 13:19:36.284045 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:36.284005    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:19:36.284336 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:36.284310    2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 14 13:19:36.284384 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:36.284368    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j" podUID="b2336252-c624-48ac-8aef-599b3ebabd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 14 13:19:37.117548 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:37.117513    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm"]
Sep 14 13:19:37.121195 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:37.121179    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm"
Sep 14 13:19:37.124190 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:37.124166    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-eff5e8-dockercfg-lrmff\""
Sep 14 13:19:37.124320 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:37.124218    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-eff5e8\""
Sep 14 13:19:37.142425 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:37.142398    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm"]
Sep 14 13:19:37.182346 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:37.182312    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b248b33-8585-4ff5-91d0-39965c7cb281-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm\" (UID: \"8b248b33-8585-4ff5-91d0-39965c7cb281\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm"
Sep 14 13:19:37.182554 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:37.182412    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vgtg\" (UniqueName: \"kubernetes.io/projected/8b248b33-8585-4ff5-91d0-39965c7cb281-kube-api-access-8vgtg\") pod \"payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm\" (UID: \"8b248b33-8585-4ff5-91d0-39965c7cb281\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm"
Sep 14 13:19:37.283422 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:37.283381    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b248b33-8585-4ff5-91d0-39965c7cb281-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm\" (UID: \"8b248b33-8585-4ff5-91d0-39965c7cb281\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm"
Sep 14 13:19:37.283613 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:37.283445    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vgtg\" (UniqueName: \"kubernetes.io/projected/8b248b33-8585-4ff5-91d0-39965c7cb281-kube-api-access-8vgtg\") pod \"payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm\" (UID: \"8b248b33-8585-4ff5-91d0-39965c7cb281\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm"
Sep 14 13:19:37.283765 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:37.283705    2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 14 13:19:37.283849 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:37.283797    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j" podUID="b2336252-c624-48ac-8aef-599b3ebabd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 14 13:19:37.284233 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:37.284210    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b248b33-8585-4ff5-91d0-39965c7cb281-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm\" (UID: \"8b248b33-8585-4ff5-91d0-39965c7cb281\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm"
Sep 14 13:19:37.295832 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:37.295807    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vgtg\" (UniqueName: \"kubernetes.io/projected/8b248b33-8585-4ff5-91d0-39965c7cb281-kube-api-access-8vgtg\") pod \"payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm\" (UID: \"8b248b33-8585-4ff5-91d0-39965c7cb281\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm"
Sep 14 13:19:37.432309 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:37.432223    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm"
Sep 14 13:19:37.745361 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:37.745332    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm"]
Sep 14 13:19:37.748096 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:19:37.747155    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b248b33_8585_4ff5_91d0_39965c7cb281.slice/crio-4e7229f65e5cefb07f6376a6a59421fb6bad3ce3df13cad7c8bd46990050e503 WatchSource:0}: Error finding container 4e7229f65e5cefb07f6376a6a59421fb6bad3ce3df13cad7c8bd46990050e503: Status 404 returned error can't find the container with id 4e7229f65e5cefb07f6376a6a59421fb6bad3ce3df13cad7c8bd46990050e503
Sep 14 13:19:38.284100 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:38.284059    2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 14 13:19:38.284490 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:38.284142    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j" podUID="b2336252-c624-48ac-8aef-599b3ebabd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 14 13:19:38.483296 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:38.483262    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm" event={"ID":"8b248b33-8585-4ff5-91d0-39965c7cb281","Type":"ContainerStarted","Data":"bb7d11560f36c162ac0c82979016c21da029323af07c7296637b5eb96d671fb0"}
Sep 14 13:19:38.483499 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:38.483312    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm" event={"ID":"8b248b33-8585-4ff5-91d0-39965c7cb281","Type":"ContainerStarted","Data":"4e7229f65e5cefb07f6376a6a59421fb6bad3ce3df13cad7c8bd46990050e503"}
Sep 14 13:19:38.483499 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:38.483344    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm"
Sep 14 13:19:38.507331 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:38.507286    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm" podStartSLOduration=1.507272414 podStartE2EDuration="1.507272414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:19:38.50509901 +0000 UTC m=+1296.939774529" watchObservedRunningTime="2026-09-14 13:19:38.507272414 +0000 UTC m=+1296.941947933"
Sep 14 13:19:39.283956 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:39.283915    2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 14 13:19:39.284265 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:39.284234    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j" podUID="b2336252-c624-48ac-8aef-599b3ebabd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 14 13:19:40.284399 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:40.284364    2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 14 13:19:40.284821 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:40.284430    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j" podUID="b2336252-c624-48ac-8aef-599b3ebabd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 14 13:19:41.336647 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:41.336620    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:19:41.337383 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:41.337355    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:19:41.338453 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:41.338434    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:19:49.493380 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:19:49.493346    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm"
Sep 14 13:20:02.257939 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:20:02.257908    2557 scope.go:117] "RemoveContainer" containerID="602ccc6b856446db23203a16b67d2b5562a689eecc06ca5f384623c5db980b8d"
Sep 14 13:21:02.281244 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:02.281209    2557 scope.go:117] "RemoveContainer" containerID="94eedaa0a0a867f52e755dca9157d07dd88b05f3eb630da02cf96debffa23695"
Sep 14 13:21:02.290526 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:02.290505    2557 scope.go:117] "RemoveContainer" containerID="cee9c2cc8b1b0a9b9f9163e08c4bc524468d0b14c1ab33cb7eed889f086cf0eb"
Sep 14 13:21:02.298816 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:02.298791    2557 scope.go:117] "RemoveContainer" containerID="346e396b0088f2d0e767cdc9263f5f3e8245916a61f19f7811130619b06c38f6"
Sep 14 13:21:02.307037 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:02.307018    2557 scope.go:117] "RemoveContainer" containerID="e5baac40f683d0f35037d9d339a978ee73a349c22b1f62a1ae0d84226a399fec"
Sep 14 13:21:02.316969 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:02.316938    2557 scope.go:117] "RemoveContainer" containerID="156fb9683365f7aa0f30fbc5fa75a073dda81cf1b439339613e5c0f55cb03fb9"
Sep 14 13:21:04.607881 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.607848    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"]
Sep 14 13:21:04.613619 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.613597    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:04.616640 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.616610    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-336ccc18\"/\"e2e-lifecycle-model-336ccc18-kserve-self-signed-certs\""
Sep 14 13:21:04.618819 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.617999    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-336ccc18\"/\"openshift-service-ca.crt\""
Sep 14 13:21:04.618819 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.618262    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-336ccc18\"/\"kube-root-ca.crt\""
Sep 14 13:21:04.619233 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.619214    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-336ccc18\"/\"default-dockercfg-pv864\""
Sep 14 13:21:04.621359 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.621339    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"]
Sep 14 13:21:04.754141 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.754103    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-tmp-dir\") pod \"e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") " pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:04.754311 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.754159    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abd45cf4-2d6a-414b-b874-6c45d77a2725-tls-certs\") pod \"e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") " pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:04.754311 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.754260    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-model-cache\") pod \"e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") " pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:04.754389 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.754345    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-dshm\") pod \"e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") " pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:04.754434 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.754401    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-home\") pod \"e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") " pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:04.754536 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.754519    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9txmg\" (UniqueName: \"kubernetes.io/projected/abd45cf4-2d6a-414b-b874-6c45d77a2725-kube-api-access-9txmg\") pod \"e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") " pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:04.855636 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.855600    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-tmp-dir\") pod \"e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") " pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:04.855636 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.855641    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abd45cf4-2d6a-414b-b874-6c45d77a2725-tls-certs\") pod \"e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") " pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:04.855904 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.855824    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-model-cache\") pod \"e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") " pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:04.855904 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.855887    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-dshm\") pod \"e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") " pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:04.856021 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.855930    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-home\") pod \"e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") " pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:04.856021 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.855989    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-tmp-dir\") pod \"e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") " pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:04.856021 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.856018    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9txmg\" (UniqueName: \"kubernetes.io/projected/abd45cf4-2d6a-414b-b874-6c45d77a2725-kube-api-access-9txmg\") pod \"e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") " pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:04.856208 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.856185    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-model-cache\") pod \"e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") " pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:04.856274 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.856258    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-home\") pod \"e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") " pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:04.857990 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.857932    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-dshm\") pod \"e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") " pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:04.858042 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.858011    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abd45cf4-2d6a-414b-b874-6c45d77a2725-tls-certs\") pod \"e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") " pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:04.864147 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.864121    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9txmg\" (UniqueName: \"kubernetes.io/projected/abd45cf4-2d6a-414b-b874-6c45d77a2725-kube-api-access-9txmg\") pod \"e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") " pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:04.925543 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:04.925506    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:05.199109 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:05.199086    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"]
Sep 14 13:21:05.200731 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:21:05.200706    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabd45cf4_2d6a_414b_b874_6c45d77a2725.slice/crio-09ac994d35988df69865c94e8e2ddbe68c7e140916142154ba02c17d2fca6129 WatchSource:0}: Error finding container 09ac994d35988df69865c94e8e2ddbe68c7e140916142154ba02c17d2fca6129: Status 404 returned error can't find the container with id 09ac994d35988df69865c94e8e2ddbe68c7e140916142154ba02c17d2fca6129
Sep 14 13:21:05.985456 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:05.985419    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r" event={"ID":"abd45cf4-2d6a-414b-b874-6c45d77a2725","Type":"ContainerStarted","Data":"df812bd7d404f5fe7fe65556f71a999e49231ec2644cdf9456d18380b77e5f33"}
Sep 14 13:21:05.985456 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:05.985458    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r" event={"ID":"abd45cf4-2d6a-414b-b874-6c45d77a2725","Type":"ContainerStarted","Data":"09ac994d35988df69865c94e8e2ddbe68c7e140916142154ba02c17d2fca6129"}
Sep 14 13:21:06.006223 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:06.006165    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r" podStartSLOduration=2.006149675 podStartE2EDuration="2.006149675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:21:06.003849241 +0000 UTC m=+1384.438524761" watchObservedRunningTime="2026-09-14 13:21:06.006149675 +0000 UTC m=+1384.440825194"
Sep 14 13:21:14.926563 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:14.926525    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:14.927083 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:14.926577    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:14.939076 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:14.939051    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:15.053397 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:15.053370    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:21:58.105549 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:21:58.105469    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"]
Sep 14 13:22:03.906175 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:03.906144    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"]
Sep 14 13:22:03.909982 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:03.909961    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:03.913048 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:03.913028    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b3d6c5\"/\"distinct-w4-b3d6c5-kserve-self-signed-certs\""
Sep 14 13:22:03.920215 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:03.920191    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"]
Sep 14 13:22:04.008611 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:04.008577    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/678f934a-1451-4a80-95d1-13bb86c9e298-tls-certs\") pod \"distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") " pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:04.008800 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:04.008615    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-home\") pod \"distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") " pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:04.008800 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:04.008644    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27jgb\" (UniqueName: \"kubernetes.io/projected/678f934a-1451-4a80-95d1-13bb86c9e298-kube-api-access-27jgb\") pod \"distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") " pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:04.008800 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:04.008704    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-dshm\") pod \"distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") " pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:04.008935 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:04.008838    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-model-cache\") pod \"distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") " pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:04.008935 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:04.008899    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-tmp-dir\") pod \"distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") " pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:04.110241 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:04.110205    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/678f934a-1451-4a80-95d1-13bb86c9e298-tls-certs\") pod \"distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") " pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:04.110241 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:04.110240    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-home\") pod \"distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") " pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:04.110465 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:04.110261    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27jgb\" (UniqueName: \"kubernetes.io/projected/678f934a-1451-4a80-95d1-13bb86c9e298-kube-api-access-27jgb\") pod \"distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") " pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:04.110465 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:04.110291    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-dshm\") pod \"distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") " pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:04.110465 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:04.110338    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-model-cache\") pod \"distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") " pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:04.110465 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:04.110406    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-tmp-dir\") pod \"distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") " pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:04.110762 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:04.110716    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-home\") pod \"distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") " pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:04.110857 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:04.110780    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-model-cache\") pod \"distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") " pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:04.110924 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:04.110891    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-tmp-dir\") pod \"distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") " pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:04.112564 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:04.112542    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-dshm\") pod \"distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") " pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:04.112715 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:04.112698    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/678f934a-1451-4a80-95d1-13bb86c9e298-tls-certs\") pod \"distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") " pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:04.118680 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:04.118658    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27jgb\" (UniqueName: \"kubernetes.io/projected/678f934a-1451-4a80-95d1-13bb86c9e298-kube-api-access-27jgb\") pod \"distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") " pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:04.221413 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:04.221391    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:04.497092 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:04.497068    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"]
Sep 14 13:22:04.498555 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:22:04.498528    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod678f934a_1451_4a80_95d1_13bb86c9e298.slice/crio-28b47b0a54efa0e24c3f8623a9f50ff70ccfb74f37512bd76cd4f869c7945c51 WatchSource:0}: Error finding container 28b47b0a54efa0e24c3f8623a9f50ff70ccfb74f37512bd76cd4f869c7945c51: Status 404 returned error can't find the container with id 28b47b0a54efa0e24c3f8623a9f50ff70ccfb74f37512bd76cd4f869c7945c51
Sep 14 13:22:05.320436 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:05.320402    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx" event={"ID":"678f934a-1451-4a80-95d1-13bb86c9e298","Type":"ContainerStarted","Data":"f6c56698cbb77612b17c3671f40e5871661792ad53df694cb042050056301f0e"}
Sep 14 13:22:05.320847 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:05.320451    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx" event={"ID":"678f934a-1451-4a80-95d1-13bb86c9e298","Type":"ContainerStarted","Data":"28b47b0a54efa0e24c3f8623a9f50ff70ccfb74f37512bd76cd4f869c7945c51"}
Sep 14 13:22:05.342281 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:05.342232    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx" podStartSLOduration=2.342217098 podStartE2EDuration="2.342217098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:22:05.340603451 +0000 UTC m=+1443.775278974" watchObservedRunningTime="2026-09-14 13:22:05.342217098 +0000 UTC m=+1443.776892646"
Sep 14 13:22:13.125526 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.125480    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v" podUID="9ee1ffa7-a413-4af3-abe9-1fe91742a1ba" containerName="main" containerID="cri-o://430547208d9810800417a9756d04b6607cb5548c1f2ed3a5ada32019dc6fac0f" gracePeriod=45
Sep 14 13:22:13.371713 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.371667    2557 generic.go:356] "Generic (PLEG): container finished" podID="9ee1ffa7-a413-4af3-abe9-1fe91742a1ba" containerID="430547208d9810800417a9756d04b6607cb5548c1f2ed3a5ada32019dc6fac0f" exitCode=0
Sep 14 13:22:13.371880 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.371774    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v" event={"ID":"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba","Type":"ContainerDied","Data":"430547208d9810800417a9756d04b6607cb5548c1f2ed3a5ada32019dc6fac0f"}
Sep 14 13:22:13.371880 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.371813    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v" event={"ID":"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba","Type":"ContainerDied","Data":"2866660ade895a7d016869dbd6008d33b00fd272505d7d4da925477e313feb6a"}
Sep 14 13:22:13.371880 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.371828    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2866660ade895a7d016869dbd6008d33b00fd272505d7d4da925477e313feb6a"
Sep 14 13:22:13.391257 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.391201    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:22:13.483558 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.483532    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-dshm\") pod \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") "
Sep 14 13:22:13.483730 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.483562    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-home\") pod \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") "
Sep 14 13:22:13.483730 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.483588    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-model-cache\") pod \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") "
Sep 14 13:22:13.483730 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.483626    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-tls-certs\") pod \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") "
Sep 14 13:22:13.483730 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.483683    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-tmp-dir\") pod \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") "
Sep 14 13:22:13.483730 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.483709    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwnx9\" (UniqueName: \"kubernetes.io/projected/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-kube-api-access-mwnx9\") pod \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\" (UID: \"9ee1ffa7-a413-4af3-abe9-1fe91742a1ba\") "
Sep 14 13:22:13.484016 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.483870    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "9ee1ffa7-a413-4af3-abe9-1fe91742a1ba" (UID: "9ee1ffa7-a413-4af3-abe9-1fe91742a1ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:22:13.484016 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.483887    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-home" (OuterVolumeSpecName: "home") pod "9ee1ffa7-a413-4af3-abe9-1fe91742a1ba" (UID: "9ee1ffa7-a413-4af3-abe9-1fe91742a1ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:22:13.484016 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.483950    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9ee1ffa7-a413-4af3-abe9-1fe91742a1ba" (UID: "9ee1ffa7-a413-4af3-abe9-1fe91742a1ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:22:13.484182 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.484170    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-home\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:22:13.484238 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.484183    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-model-cache\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:22:13.484238 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.484192    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-tmp-dir\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:22:13.485828 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.485803    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-dshm" (OuterVolumeSpecName: "dshm") pod "9ee1ffa7-a413-4af3-abe9-1fe91742a1ba" (UID: "9ee1ffa7-a413-4af3-abe9-1fe91742a1ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:22:13.485930 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.485888    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ee1ffa7-a413-4af3-abe9-1fe91742a1ba" (UID: "9ee1ffa7-a413-4af3-abe9-1fe91742a1ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:22:13.485930 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.485912    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-kube-api-access-mwnx9" (OuterVolumeSpecName: "kube-api-access-mwnx9") pod "9ee1ffa7-a413-4af3-abe9-1fe91742a1ba" (UID: "9ee1ffa7-a413-4af3-abe9-1fe91742a1ba"). InnerVolumeSpecName "kube-api-access-mwnx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:22:13.585556 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.585528    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-tls-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:22:13.585556 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.585552    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwnx9\" (UniqueName: \"kubernetes.io/projected/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-kube-api-access-mwnx9\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:22:13.585556 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:13.585561    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba-dshm\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:22:14.221870 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:14.221831    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:14.222291 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:14.221888    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:14.234585 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:14.234563    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:14.377684 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:14.377606    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"
Sep 14 13:22:14.392206 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:14.392182    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:22:14.396198 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:14.396169    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"]
Sep 14 13:22:14.400330 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:14.400305    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b3d6c5/e2e-conflict-a-c95ce5f6-kserve-5dcccc9c95-wgq9v"]
Sep 14 13:22:16.049453 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:16.049421    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ee1ffa7-a413-4af3-abe9-1fe91742a1ba" path="/var/lib/kubelet/pods/9ee1ffa7-a413-4af3-abe9-1fe91742a1ba/volumes"
Sep 14 13:22:31.519366 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:31.519332    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-336ccc18-n5ctw"]
Sep 14 13:22:31.519969 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:31.519950    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ee1ffa7-a413-4af3-abe9-1fe91742a1ba" containerName="main"
Sep 14 13:22:31.520051 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:31.519971    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ee1ffa7-a413-4af3-abe9-1fe91742a1ba" containerName="main"
Sep 14 13:22:31.520107 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:31.520076    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ee1ffa7-a413-4af3-abe9-1fe91742a1ba" containerName="main"
Sep 14 13:22:31.528250 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:31.528221    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-336ccc18-n5ctw"
Sep 14 13:22:31.531067 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:31.531041    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbzp\""
Sep 14 13:22:31.531067 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:31.531057    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-336ccc18-n5ctw"]
Sep 14 13:22:31.663259 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:31.663227    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca51e17a-edd3-4bc5-a887-1b4ef630b3c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-336ccc18-n5ctw\" (UID: \"ca51e17a-edd3-4bc5-a887-1b4ef630b3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-336ccc18-n5ctw"
Sep 14 13:22:31.764631 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:31.764598    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca51e17a-edd3-4bc5-a887-1b4ef630b3c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-336ccc18-n5ctw\" (UID: \"ca51e17a-edd3-4bc5-a887-1b4ef630b3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-336ccc18-n5ctw"
Sep 14 13:22:31.765340 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:31.765313    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca51e17a-edd3-4bc5-a887-1b4ef630b3c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-336ccc18-n5ctw\" (UID: \"ca51e17a-edd3-4bc5-a887-1b4ef630b3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-336ccc18-n5ctw"
Sep 14 13:22:31.841893 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:31.841815    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-336ccc18-n5ctw"
Sep 14 13:22:32.118729 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:32.118702    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-336ccc18-n5ctw"]
Sep 14 13:22:32.120323 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:22:32.120297    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca51e17a_edd3_4bc5_a887_1b4ef630b3c4.slice/crio-15f477e5a58fbc88d7293b62d578d6fafc7c8914c221058145b30a98f4964359 WatchSource:0}: Error finding container 15f477e5a58fbc88d7293b62d578d6fafc7c8914c221058145b30a98f4964359: Status 404 returned error can't find the container with id 15f477e5a58fbc88d7293b62d578d6fafc7c8914c221058145b30a98f4964359
Sep 14 13:22:32.484760 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:32.484701    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-336ccc18-n5ctw" event={"ID":"ca51e17a-edd3-4bc5-a887-1b4ef630b3c4","Type":"ContainerStarted","Data":"beb0921f5e17bf627f66d6ed0f79d3dc3b211d0713b04ac77783137c68078792"}
Sep 14 13:22:32.484969 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:32.484768    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-336ccc18-n5ctw" event={"ID":"ca51e17a-edd3-4bc5-a887-1b4ef630b3c4","Type":"ContainerStarted","Data":"15f477e5a58fbc88d7293b62d578d6fafc7c8914c221058145b30a98f4964359"}
Sep 14 13:22:32.507818 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:32.507769    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-336ccc18-n5ctw" podStartSLOduration=1.507732847 podStartE2EDuration="1.507732847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:22:32.504861857 +0000 UTC m=+1470.939537386" watchObservedRunningTime="2026-09-14 13:22:32.507732847 +0000 UTC m=+1470.942408366"
Sep 14 13:22:33.492513 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:33.492477    2557 generic.go:356] "Generic (PLEG): container finished" podID="ca51e17a-edd3-4bc5-a887-1b4ef630b3c4" containerID="beb0921f5e17bf627f66d6ed0f79d3dc3b211d0713b04ac77783137c68078792" exitCode=0
Sep 14 13:22:33.492949 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:33.492538    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-336ccc18-n5ctw" event={"ID":"ca51e17a-edd3-4bc5-a887-1b4ef630b3c4","Type":"ContainerDied","Data":"beb0921f5e17bf627f66d6ed0f79d3dc3b211d0713b04ac77783137c68078792"}
Sep 14 13:22:34.640063 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:34.640038    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-336ccc18-n5ctw"
Sep 14 13:22:34.793760 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:34.793663    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca51e17a-edd3-4bc5-a887-1b4ef630b3c4-maas-api-service-ca\") pod \"ca51e17a-edd3-4bc5-a887-1b4ef630b3c4\" (UID: \"ca51e17a-edd3-4bc5-a887-1b4ef630b3c4\") "
Sep 14 13:22:34.794022 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:34.794000    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca51e17a-edd3-4bc5-a887-1b4ef630b3c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ca51e17a-edd3-4bc5-a887-1b4ef630b3c4" (UID: "ca51e17a-edd3-4bc5-a887-1b4ef630b3c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:22:34.894684 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:34.894653    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca51e17a-edd3-4bc5-a887-1b4ef630b3c4-maas-api-service-ca\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:22:35.505894 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:35.505863    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-336ccc18-n5ctw"
Sep 14 13:22:35.505894 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:35.505879    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-336ccc18-n5ctw" event={"ID":"ca51e17a-edd3-4bc5-a887-1b4ef630b3c4","Type":"ContainerDied","Data":"15f477e5a58fbc88d7293b62d578d6fafc7c8914c221058145b30a98f4964359"}
Sep 14 13:22:35.506099 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:35.505968    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15f477e5a58fbc88d7293b62d578d6fafc7c8914c221058145b30a98f4964359"
Sep 14 13:22:47.536711 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:47.536667    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-336ccc18-n5ctw"]
Sep 14 13:22:47.545110 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:47.545082    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-336ccc18-n5ctw"]
Sep 14 13:22:48.049820 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:48.049789    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca51e17a-edd3-4bc5-a887-1b4ef630b3c4" path="/var/lib/kubelet/pods/ca51e17a-edd3-4bc5-a887-1b4ef630b3c4/volumes"
Sep 14 13:22:55.799611 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:22:55.799574    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"]
Sep 14 13:23:02.128024 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:02.127988    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwqnf_62d953d6-3b86-4c95-9a47-9579d88993a7/ovn-acl-logging/0.log"
Sep 14 13:23:02.128501 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:02.128483    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwqnf_62d953d6-3b86-4c95-9a47-9579d88993a7/ovn-acl-logging/0.log"
Sep 14 13:23:10.818826 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:10.818783    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r" podUID="abd45cf4-2d6a-414b-b874-6c45d77a2725" containerName="main" containerID="cri-o://df812bd7d404f5fe7fe65556f71a999e49231ec2644cdf9456d18380b77e5f33" gracePeriod=45
Sep 14 13:23:11.085144 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.085115    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:23:11.128280 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.128253    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9txmg\" (UniqueName: \"kubernetes.io/projected/abd45cf4-2d6a-414b-b874-6c45d77a2725-kube-api-access-9txmg\") pod \"abd45cf4-2d6a-414b-b874-6c45d77a2725\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") "
Sep 14 13:23:11.128460 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.128291    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-model-cache\") pod \"abd45cf4-2d6a-414b-b874-6c45d77a2725\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") "
Sep 14 13:23:11.128460 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.128320    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-dshm\") pod \"abd45cf4-2d6a-414b-b874-6c45d77a2725\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") "
Sep 14 13:23:11.128460 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.128370    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abd45cf4-2d6a-414b-b874-6c45d77a2725-tls-certs\") pod \"abd45cf4-2d6a-414b-b874-6c45d77a2725\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") "
Sep 14 13:23:11.128460 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.128445    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-tmp-dir\") pod \"abd45cf4-2d6a-414b-b874-6c45d77a2725\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") "
Sep 14 13:23:11.128682 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.128505    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-home\") pod \"abd45cf4-2d6a-414b-b874-6c45d77a2725\" (UID: \"abd45cf4-2d6a-414b-b874-6c45d77a2725\") "
Sep 14 13:23:11.128682 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.128508    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-model-cache" (OuterVolumeSpecName: "model-cache") pod "abd45cf4-2d6a-414b-b874-6c45d77a2725" (UID: "abd45cf4-2d6a-414b-b874-6c45d77a2725"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:23:11.128927 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.128896    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "abd45cf4-2d6a-414b-b874-6c45d77a2725" (UID: "abd45cf4-2d6a-414b-b874-6c45d77a2725"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:23:11.129054 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.128920    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-model-cache\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:23:11.129365 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.129337    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-home" (OuterVolumeSpecName: "home") pod "abd45cf4-2d6a-414b-b874-6c45d77a2725" (UID: "abd45cf4-2d6a-414b-b874-6c45d77a2725"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:23:11.130817 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.130716    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abd45cf4-2d6a-414b-b874-6c45d77a2725-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "abd45cf4-2d6a-414b-b874-6c45d77a2725" (UID: "abd45cf4-2d6a-414b-b874-6c45d77a2725"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:23:11.130817 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.130773    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd45cf4-2d6a-414b-b874-6c45d77a2725-kube-api-access-9txmg" (OuterVolumeSpecName: "kube-api-access-9txmg") pod "abd45cf4-2d6a-414b-b874-6c45d77a2725" (UID: "abd45cf4-2d6a-414b-b874-6c45d77a2725"). InnerVolumeSpecName "kube-api-access-9txmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:23:11.130978 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.130922    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-dshm" (OuterVolumeSpecName: "dshm") pod "abd45cf4-2d6a-414b-b874-6c45d77a2725" (UID: "abd45cf4-2d6a-414b-b874-6c45d77a2725"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:23:11.229963 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.229930    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-home\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:23:11.229963 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.229957    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9txmg\" (UniqueName: \"kubernetes.io/projected/abd45cf4-2d6a-414b-b874-6c45d77a2725-kube-api-access-9txmg\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:23:11.229963 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.229970    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-dshm\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:23:11.230189 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.229981    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abd45cf4-2d6a-414b-b874-6c45d77a2725-tls-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:23:11.230189 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.229992    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abd45cf4-2d6a-414b-b874-6c45d77a2725-tmp-dir\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:23:11.713118 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.713084    2557 generic.go:356] "Generic (PLEG): container finished" podID="abd45cf4-2d6a-414b-b874-6c45d77a2725" containerID="df812bd7d404f5fe7fe65556f71a999e49231ec2644cdf9456d18380b77e5f33" exitCode=0
Sep 14 13:23:11.713297 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.713131    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"
Sep 14 13:23:11.713297 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.713165    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r" event={"ID":"abd45cf4-2d6a-414b-b874-6c45d77a2725","Type":"ContainerDied","Data":"df812bd7d404f5fe7fe65556f71a999e49231ec2644cdf9456d18380b77e5f33"}
Sep 14 13:23:11.713297 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.713210    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r" event={"ID":"abd45cf4-2d6a-414b-b874-6c45d77a2725","Type":"ContainerDied","Data":"09ac994d35988df69865c94e8e2ddbe68c7e140916142154ba02c17d2fca6129"}
Sep 14 13:23:11.713297 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.713227    2557 scope.go:117] "RemoveContainer" containerID="df812bd7d404f5fe7fe65556f71a999e49231ec2644cdf9456d18380b77e5f33"
Sep 14 13:23:11.722768 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.722724    2557 scope.go:117] "RemoveContainer" containerID="df812bd7d404f5fe7fe65556f71a999e49231ec2644cdf9456d18380b77e5f33"
Sep 14 13:23:11.723049 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:23:11.723031    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df812bd7d404f5fe7fe65556f71a999e49231ec2644cdf9456d18380b77e5f33\": container with ID starting with df812bd7d404f5fe7fe65556f71a999e49231ec2644cdf9456d18380b77e5f33 not found: ID does not exist" containerID="df812bd7d404f5fe7fe65556f71a999e49231ec2644cdf9456d18380b77e5f33"
Sep 14 13:23:11.723115 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.723061    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df812bd7d404f5fe7fe65556f71a999e49231ec2644cdf9456d18380b77e5f33"} err="failed to get container status \"df812bd7d404f5fe7fe65556f71a999e49231ec2644cdf9456d18380b77e5f33\": rpc error: code = NotFound desc = could not find container \"df812bd7d404f5fe7fe65556f71a999e49231ec2644cdf9456d18380b77e5f33\": container with ID starting with df812bd7d404f5fe7fe65556f71a999e49231ec2644cdf9456d18380b77e5f33 not found: ID does not exist"
Sep 14 13:23:11.738609 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.738583    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"]
Sep 14 13:23:11.742956 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:11.742931    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-336ccc18/e2e-lifecycle-model-336ccc18-kserve-6fff4f79cf-w474r"]
Sep 14 13:23:12.049925 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:12.049842    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abd45cf4-2d6a-414b-b874-6c45d77a2725" path="/var/lib/kubelet/pods/abd45cf4-2d6a-414b-b874-6c45d77a2725/volumes"
Sep 14 13:23:29.361256 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.361180    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"]
Sep 14 13:23:29.361685 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.361591    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abd45cf4-2d6a-414b-b874-6c45d77a2725" containerName="main"
Sep 14 13:23:29.361685 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.361603    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="abd45cf4-2d6a-414b-b874-6c45d77a2725" containerName="main"
Sep 14 13:23:29.361685 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.361628    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca51e17a-edd3-4bc5-a887-1b4ef630b3c4" containerName="revoke-keys"
Sep 14 13:23:29.361685 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.361634    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca51e17a-edd3-4bc5-a887-1b4ef630b3c4" containerName="revoke-keys"
Sep 14 13:23:29.361841 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.361698    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca51e17a-edd3-4bc5-a887-1b4ef630b3c4" containerName="revoke-keys"
Sep 14 13:23:29.361841 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.361709    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="abd45cf4-2d6a-414b-b874-6c45d77a2725" containerName="main"
Sep 14 13:23:29.364448 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.364426    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:29.368299 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.368279    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-bc851d\"/\"embedding-w0-bc851d-kserve-self-signed-certs\""
Sep 14 13:23:29.385314 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.385290    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"]
Sep 14 13:23:29.507367 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.507336    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-tls-certs\") pod \"embedding-w0-bc851d-kserve-769f4795c5-p5c7f\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") " pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:29.507533 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.507398    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr8r4\" (UniqueName: \"kubernetes.io/projected/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-kube-api-access-zr8r4\") pod \"embedding-w0-bc851d-kserve-769f4795c5-p5c7f\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") " pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:29.507533 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.507458    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-model-cache\") pod \"embedding-w0-bc851d-kserve-769f4795c5-p5c7f\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") " pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:29.507533 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.507484    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-dshm\") pod \"embedding-w0-bc851d-kserve-769f4795c5-p5c7f\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") " pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:29.507533 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.507511    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-tmp-dir\") pod \"embedding-w0-bc851d-kserve-769f4795c5-p5c7f\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") " pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:29.507685 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.507550    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-home\") pod \"embedding-w0-bc851d-kserve-769f4795c5-p5c7f\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") " pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:29.608591 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.608554    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-tls-certs\") pod \"embedding-w0-bc851d-kserve-769f4795c5-p5c7f\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") " pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:29.608808 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.608612    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr8r4\" (UniqueName: \"kubernetes.io/projected/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-kube-api-access-zr8r4\") pod \"embedding-w0-bc851d-kserve-769f4795c5-p5c7f\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") " pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:29.608808 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.608638    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-model-cache\") pod \"embedding-w0-bc851d-kserve-769f4795c5-p5c7f\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") " pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:29.608808 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.608660    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-dshm\") pod \"embedding-w0-bc851d-kserve-769f4795c5-p5c7f\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") " pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:29.608808 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.608675    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-tmp-dir\") pod \"embedding-w0-bc851d-kserve-769f4795c5-p5c7f\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") " pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:29.608808 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.608722    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-home\") pod \"embedding-w0-bc851d-kserve-769f4795c5-p5c7f\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") " pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:29.609134 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.609113    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-model-cache\") pod \"embedding-w0-bc851d-kserve-769f4795c5-p5c7f\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") " pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:29.609198 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.609118    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-home\") pod \"embedding-w0-bc851d-kserve-769f4795c5-p5c7f\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") " pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:29.609245 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.609192    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-tmp-dir\") pod \"embedding-w0-bc851d-kserve-769f4795c5-p5c7f\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") " pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:29.611013 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.610988    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-dshm\") pod \"embedding-w0-bc851d-kserve-769f4795c5-p5c7f\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") " pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:29.611297 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.611260    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-tls-certs\") pod \"embedding-w0-bc851d-kserve-769f4795c5-p5c7f\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") " pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:29.620468 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.620448    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr8r4\" (UniqueName: \"kubernetes.io/projected/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-kube-api-access-zr8r4\") pod \"embedding-w0-bc851d-kserve-769f4795c5-p5c7f\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") " pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:29.673589 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.673559    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:29.952770 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:29.952731    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"]
Sep 14 13:23:29.954702 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:23:29.954675    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f52a7f4_8cef_44d2_8f72_d8cf9c62a35c.slice/crio-14f755a0d131d66588ab6f275bc2cc65c0549a8b47af057a8417928dab160204 WatchSource:0}: Error finding container 14f755a0d131d66588ab6f275bc2cc65c0549a8b47af057a8417928dab160204: Status 404 returned error can't find the container with id 14f755a0d131d66588ab6f275bc2cc65c0549a8b47af057a8417928dab160204
Sep 14 13:23:30.820340 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:30.820302    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f" event={"ID":"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c","Type":"ContainerStarted","Data":"455e8b9ccef9f1bb7017fc686d03033946116751b403479ad85f194a1fc3fae8"}
Sep 14 13:23:30.820734 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:30.820351    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f" event={"ID":"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c","Type":"ContainerStarted","Data":"14f755a0d131d66588ab6f275bc2cc65c0549a8b47af057a8417928dab160204"}
Sep 14 13:23:30.845769 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:30.845709    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f" podStartSLOduration=1.845693114 podStartE2EDuration="1.845693114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:23:30.844536882 +0000 UTC m=+1529.279212401" watchObservedRunningTime="2026-09-14 13:23:30.845693114 +0000 UTC m=+1529.280368633"
Sep 14 13:23:39.674391 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:39.674358    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:39.674940 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:39.674507    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:39.687355 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:39.687327    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:23:39.883109 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:23:39.883079    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:24:02.368078 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:24:02.368048    2557 scope.go:117] "RemoveContainer" containerID="1bacffdbe7c0e5cbbeba637687dfbdbc93b2670a772e85ec03366a309144d7dd"
Sep 14 13:24:02.378853 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:24:02.378834    2557 scope.go:117] "RemoveContainer" containerID="f9bc123b34ec7bba30874180af9631edbaa265b940428a1a0ac2b7172d1d4084"
Sep 14 13:24:02.386828 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:24:02.386808    2557 scope.go:117] "RemoveContainer" containerID="430547208d9810800417a9756d04b6607cb5548c1f2ed3a5ada32019dc6fac0f"
Sep 14 13:25:02.418336 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:25:02.418250    2557 scope.go:117] "RemoveContainer" containerID="4095ec93a0e0cc0f0744279a4c53f24ad8ed1efbd46df7c9f918eba731503e88"
Sep 14 13:25:02.426940 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:25:02.426918    2557 scope.go:117] "RemoveContainer" containerID="f84091a8b1e1c5d4d817e6ed1c6b254bc24147a005621790a6c7e147d6ed611d"
Sep 14 13:26:22.256733 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:22.256698    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc851d-wjs7d"]
Sep 14 13:26:22.260337 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:22.260320    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3d6c5-qbwk5"]
Sep 14 13:26:22.260457 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:22.260437    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc851d-wjs7d"
Sep 14 13:26:22.263198 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:22.263176    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbzp\""
Sep 14 13:26:22.263747 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:22.263718    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3d6c5-qbwk5"
Sep 14 13:26:22.273786 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:22.273765    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc851d-wjs7d"]
Sep 14 13:26:22.276852 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:22.276829    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3d6c5-qbwk5"]
Sep 14 13:26:22.395368 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:22.395337    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d23896d-7a76-43cb-ab13-b49d79f23edc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b3d6c5-qbwk5\" (UID: \"9d23896d-7a76-43cb-ab13-b49d79f23edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3d6c5-qbwk5"
Sep 14 13:26:22.395537 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:22.395469    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e0bf3bf-6203-4820-8b6f-9029792fefc7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-bc851d-wjs7d\" (UID: \"4e0bf3bf-6203-4820-8b6f-9029792fefc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc851d-wjs7d"
Sep 14 13:26:22.496069 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:22.496035    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e0bf3bf-6203-4820-8b6f-9029792fefc7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-bc851d-wjs7d\" (UID: \"4e0bf3bf-6203-4820-8b6f-9029792fefc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc851d-wjs7d"
Sep 14 13:26:22.496264 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:22.496092    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d23896d-7a76-43cb-ab13-b49d79f23edc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b3d6c5-qbwk5\" (UID: \"9d23896d-7a76-43cb-ab13-b49d79f23edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3d6c5-qbwk5"
Sep 14 13:26:22.496813 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:22.496788    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e0bf3bf-6203-4820-8b6f-9029792fefc7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-bc851d-wjs7d\" (UID: \"4e0bf3bf-6203-4820-8b6f-9029792fefc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc851d-wjs7d"
Sep 14 13:26:22.496933 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:22.496823    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d23896d-7a76-43cb-ab13-b49d79f23edc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b3d6c5-qbwk5\" (UID: \"9d23896d-7a76-43cb-ab13-b49d79f23edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3d6c5-qbwk5"
Sep 14 13:26:22.573622 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:22.573544    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc851d-wjs7d"
Sep 14 13:26:22.581623 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:22.581603    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3d6c5-qbwk5"
Sep 14 13:26:22.848328 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:22.848301    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc851d-wjs7d"]
Sep 14 13:26:22.853207 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:26:22.852841    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0bf3bf_6203_4820_8b6f_9029792fefc7.slice/crio-1beae65040fe299fbbf4d991e64dc7dd4504aa7c74ca458f8310c34a4d0d87e4 WatchSource:0}: Error finding container 1beae65040fe299fbbf4d991e64dc7dd4504aa7c74ca458f8310c34a4d0d87e4: Status 404 returned error can't find the container with id 1beae65040fe299fbbf4d991e64dc7dd4504aa7c74ca458f8310c34a4d0d87e4
Sep 14 13:26:22.862611 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:22.862586    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3d6c5-qbwk5"]
Sep 14 13:26:22.866382 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:26:22.866359    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d23896d_7a76_43cb_ab13_b49d79f23edc.slice/crio-693cfda52405e6ece931344f6db7cdd034d23b10fb1f99eed8ab11ee371ef744 WatchSource:0}: Error finding container 693cfda52405e6ece931344f6db7cdd034d23b10fb1f99eed8ab11ee371ef744: Status 404 returned error can't find the container with id 693cfda52405e6ece931344f6db7cdd034d23b10fb1f99eed8ab11ee371ef744
Sep 14 13:26:23.786764 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:23.786714    2557 generic.go:356] "Generic (PLEG): container finished" podID="4e0bf3bf-6203-4820-8b6f-9029792fefc7" containerID="cd2ab91df0c7df70271748536c685de485ee071e1025729d9bf231633fd67156" exitCode=0
Sep 14 13:26:23.787190 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:23.786803    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc851d-wjs7d" event={"ID":"4e0bf3bf-6203-4820-8b6f-9029792fefc7","Type":"ContainerDied","Data":"cd2ab91df0c7df70271748536c685de485ee071e1025729d9bf231633fd67156"}
Sep 14 13:26:23.787190 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:23.786851    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc851d-wjs7d" event={"ID":"4e0bf3bf-6203-4820-8b6f-9029792fefc7","Type":"ContainerStarted","Data":"1beae65040fe299fbbf4d991e64dc7dd4504aa7c74ca458f8310c34a4d0d87e4"}
Sep 14 13:26:23.788544 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:23.788524    2557 generic.go:356] "Generic (PLEG): container finished" podID="9d23896d-7a76-43cb-ab13-b49d79f23edc" containerID="425d26617440772fe96991df084778cb6f11d5de5a833bac7bad5b3afad5213d" exitCode=0
Sep 14 13:26:23.788644 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:23.788552    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3d6c5-qbwk5" event={"ID":"9d23896d-7a76-43cb-ab13-b49d79f23edc","Type":"ContainerDied","Data":"425d26617440772fe96991df084778cb6f11d5de5a833bac7bad5b3afad5213d"}
Sep 14 13:26:23.788644 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:23.788579    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3d6c5-qbwk5" event={"ID":"9d23896d-7a76-43cb-ab13-b49d79f23edc","Type":"ContainerStarted","Data":"693cfda52405e6ece931344f6db7cdd034d23b10fb1f99eed8ab11ee371ef744"}
Sep 14 13:26:24.933701 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:24.933678    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3d6c5-qbwk5"
Sep 14 13:26:24.962521 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:24.962494    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc851d-wjs7d"
Sep 14 13:26:25.020021 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:25.019988    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d23896d-7a76-43cb-ab13-b49d79f23edc-maas-api-service-ca\") pod \"9d23896d-7a76-43cb-ab13-b49d79f23edc\" (UID: \"9d23896d-7a76-43cb-ab13-b49d79f23edc\") "
Sep 14 13:26:25.020344 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:25.020322    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d23896d-7a76-43cb-ab13-b49d79f23edc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9d23896d-7a76-43cb-ab13-b49d79f23edc" (UID: "9d23896d-7a76-43cb-ab13-b49d79f23edc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:26:25.121236 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:25.121143    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e0bf3bf-6203-4820-8b6f-9029792fefc7-maas-api-service-ca\") pod \"4e0bf3bf-6203-4820-8b6f-9029792fefc7\" (UID: \"4e0bf3bf-6203-4820-8b6f-9029792fefc7\") "
Sep 14 13:26:25.121793 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:25.121508    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e0bf3bf-6203-4820-8b6f-9029792fefc7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e0bf3bf-6203-4820-8b6f-9029792fefc7" (UID: "4e0bf3bf-6203-4820-8b6f-9029792fefc7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:26:25.121793 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:25.121636    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e0bf3bf-6203-4820-8b6f-9029792fefc7-maas-api-service-ca\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:25.121793 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:25.121654    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d23896d-7a76-43cb-ab13-b49d79f23edc-maas-api-service-ca\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:25.801996 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:25.801968    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc851d-wjs7d"
Sep 14 13:26:25.801996 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:25.801972    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc851d-wjs7d" event={"ID":"4e0bf3bf-6203-4820-8b6f-9029792fefc7","Type":"ContainerDied","Data":"1beae65040fe299fbbf4d991e64dc7dd4504aa7c74ca458f8310c34a4d0d87e4"}
Sep 14 13:26:25.802232 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:25.802022    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1beae65040fe299fbbf4d991e64dc7dd4504aa7c74ca458f8310c34a4d0d87e4"
Sep 14 13:26:25.803453 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:25.803425    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3d6c5-qbwk5" event={"ID":"9d23896d-7a76-43cb-ab13-b49d79f23edc","Type":"ContainerDied","Data":"693cfda52405e6ece931344f6db7cdd034d23b10fb1f99eed8ab11ee371ef744"}
Sep 14 13:26:25.803555 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:25.803473    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="693cfda52405e6ece931344f6db7cdd034d23b10fb1f99eed8ab11ee371ef744"
Sep 14 13:26:25.803555 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:25.803435    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3d6c5-qbwk5"
Sep 14 13:26:26.460451 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:26.460420    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"]
Sep 14 13:26:26.469183 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:26.469157    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"]
Sep 14 13:26:26.495346 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:26.495313    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"]
Sep 14 13:26:26.511039 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:26.511012    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"]
Sep 14 13:26:34.339938 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.339903    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc851d-29823195-t4bjm"]
Sep 14 13:26:34.342318 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.342288    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc851d-29823195-t4bjm"]
Sep 14 13:26:34.367531 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.367496    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4"]
Sep 14 13:26:34.367803 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.367772    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4" podUID="2d3c87e8-36a9-4212-89b4-7e1ba1b64b93" containerName="payload-processing" containerID="cri-o://a2843489ca9fdb173b509462c191f34b6d176397761ce07563d77f23d389c71f" gracePeriod=30
Sep 14 13:26:34.377706 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.377675    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877"]
Sep 14 13:26:34.378156 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.378074    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877" podUID="b7b603c0-b71c-4631-b269-bbd00ca584fe" containerName="payload-pre-processing" containerID="cri-o://b61febad8587bc45b9251c477e2fa7634ce24eefffdabc2a499dbaeb2d5d3e75" gracePeriod=30
Sep 14 13:26:34.553179 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.552186    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc851d-wjs7d"]
Sep 14 13:26:34.555808 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.555766    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc851d-wjs7d"]
Sep 14 13:26:34.666867 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.666840    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4"
Sep 14 13:26:34.671161 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.671136    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877"
Sep 14 13:26:34.820106 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.820047    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3c87e8-36a9-4212-89b4-7e1ba1b64b93-plugins-config-volume\") pod \"2d3c87e8-36a9-4212-89b4-7e1ba1b64b93\" (UID: \"2d3c87e8-36a9-4212-89b4-7e1ba1b64b93\") "
Sep 14 13:26:34.820106 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.820104    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7b603c0-b71c-4631-b269-bbd00ca584fe-plugins-config-volume\") pod \"b7b603c0-b71c-4631-b269-bbd00ca584fe\" (UID: \"b7b603c0-b71c-4631-b269-bbd00ca584fe\") "
Sep 14 13:26:34.820374 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.820174    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9lrr\" (UniqueName: \"kubernetes.io/projected/2d3c87e8-36a9-4212-89b4-7e1ba1b64b93-kube-api-access-s9lrr\") pod \"2d3c87e8-36a9-4212-89b4-7e1ba1b64b93\" (UID: \"2d3c87e8-36a9-4212-89b4-7e1ba1b64b93\") "
Sep 14 13:26:34.820374 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.820203    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddm7c\" (UniqueName: \"kubernetes.io/projected/b7b603c0-b71c-4631-b269-bbd00ca584fe-kube-api-access-ddm7c\") pod \"b7b603c0-b71c-4631-b269-bbd00ca584fe\" (UID: \"b7b603c0-b71c-4631-b269-bbd00ca584fe\") "
Sep 14 13:26:34.820536 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.820479    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7b603c0-b71c-4631-b269-bbd00ca584fe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7b603c0-b71c-4631-b269-bbd00ca584fe" (UID: "b7b603c0-b71c-4631-b269-bbd00ca584fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:26:34.820536 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.820523    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d3c87e8-36a9-4212-89b4-7e1ba1b64b93-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2d3c87e8-36a9-4212-89b4-7e1ba1b64b93" (UID: "2d3c87e8-36a9-4212-89b4-7e1ba1b64b93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:26:34.820695 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.820678    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3c87e8-36a9-4212-89b4-7e1ba1b64b93-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:34.820764 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.820700    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7b603c0-b71c-4631-b269-bbd00ca584fe-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:34.822319 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.822296    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3c87e8-36a9-4212-89b4-7e1ba1b64b93-kube-api-access-s9lrr" (OuterVolumeSpecName: "kube-api-access-s9lrr") pod "2d3c87e8-36a9-4212-89b4-7e1ba1b64b93" (UID: "2d3c87e8-36a9-4212-89b4-7e1ba1b64b93"). InnerVolumeSpecName "kube-api-access-s9lrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:26:34.822433 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.822338    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7b603c0-b71c-4631-b269-bbd00ca584fe-kube-api-access-ddm7c" (OuterVolumeSpecName: "kube-api-access-ddm7c") pod "b7b603c0-b71c-4631-b269-bbd00ca584fe" (UID: "b7b603c0-b71c-4631-b269-bbd00ca584fe"). InnerVolumeSpecName "kube-api-access-ddm7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:26:34.859075 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.859040    2557 generic.go:356] "Generic (PLEG): container finished" podID="b7b603c0-b71c-4631-b269-bbd00ca584fe" containerID="b61febad8587bc45b9251c477e2fa7634ce24eefffdabc2a499dbaeb2d5d3e75" exitCode=0
Sep 14 13:26:34.859225 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.859080    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877"
Sep 14 13:26:34.859225 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.859118    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877" event={"ID":"b7b603c0-b71c-4631-b269-bbd00ca584fe","Type":"ContainerDied","Data":"b61febad8587bc45b9251c477e2fa7634ce24eefffdabc2a499dbaeb2d5d3e75"}
Sep 14 13:26:34.859225 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.859173    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877" event={"ID":"b7b603c0-b71c-4631-b269-bbd00ca584fe","Type":"ContainerDied","Data":"7f6329388207702251cc120b3a2be0c1913afc5e16a928bc2962eaad51036cca"}
Sep 14 13:26:34.859225 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.859199    2557 scope.go:117] "RemoveContainer" containerID="b61febad8587bc45b9251c477e2fa7634ce24eefffdabc2a499dbaeb2d5d3e75"
Sep 14 13:26:34.860783 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.860760    2557 generic.go:356] "Generic (PLEG): container finished" podID="2d3c87e8-36a9-4212-89b4-7e1ba1b64b93" containerID="a2843489ca9fdb173b509462c191f34b6d176397761ce07563d77f23d389c71f" exitCode=0
Sep 14 13:26:34.860916 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.860785    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4" event={"ID":"2d3c87e8-36a9-4212-89b4-7e1ba1b64b93","Type":"ContainerDied","Data":"a2843489ca9fdb173b509462c191f34b6d176397761ce07563d77f23d389c71f"}
Sep 14 13:26:34.860916 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.860806    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4"
Sep 14 13:26:34.860916 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.860826    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4" event={"ID":"2d3c87e8-36a9-4212-89b4-7e1ba1b64b93","Type":"ContainerDied","Data":"a379b9aa44e9dc312ff3206e79dd695e862bc34e8a1e5ee985b70795735cebbe"}
Sep 14 13:26:34.870167 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.870152    2557 scope.go:117] "RemoveContainer" containerID="b61febad8587bc45b9251c477e2fa7634ce24eefffdabc2a499dbaeb2d5d3e75"
Sep 14 13:26:34.870470 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:26:34.870452    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b61febad8587bc45b9251c477e2fa7634ce24eefffdabc2a499dbaeb2d5d3e75\": container with ID starting with b61febad8587bc45b9251c477e2fa7634ce24eefffdabc2a499dbaeb2d5d3e75 not found: ID does not exist" containerID="b61febad8587bc45b9251c477e2fa7634ce24eefffdabc2a499dbaeb2d5d3e75"
Sep 14 13:26:34.870519 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.870478    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b61febad8587bc45b9251c477e2fa7634ce24eefffdabc2a499dbaeb2d5d3e75"} err="failed to get container status \"b61febad8587bc45b9251c477e2fa7634ce24eefffdabc2a499dbaeb2d5d3e75\": rpc error: code = NotFound desc = could not find container \"b61febad8587bc45b9251c477e2fa7634ce24eefffdabc2a499dbaeb2d5d3e75\": container with ID starting with b61febad8587bc45b9251c477e2fa7634ce24eefffdabc2a499dbaeb2d5d3e75 not found: ID does not exist"
Sep 14 13:26:34.870519 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.870494    2557 scope.go:117] "RemoveContainer" containerID="a2843489ca9fdb173b509462c191f34b6d176397761ce07563d77f23d389c71f"
Sep 14 13:26:34.881367 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.881350    2557 scope.go:117] "RemoveContainer" containerID="a2843489ca9fdb173b509462c191f34b6d176397761ce07563d77f23d389c71f"
Sep 14 13:26:34.881688 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:26:34.881667    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2843489ca9fdb173b509462c191f34b6d176397761ce07563d77f23d389c71f\": container with ID starting with a2843489ca9fdb173b509462c191f34b6d176397761ce07563d77f23d389c71f not found: ID does not exist" containerID="a2843489ca9fdb173b509462c191f34b6d176397761ce07563d77f23d389c71f"
Sep 14 13:26:34.881812 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.881694    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2843489ca9fdb173b509462c191f34b6d176397761ce07563d77f23d389c71f"} err="failed to get container status \"a2843489ca9fdb173b509462c191f34b6d176397761ce07563d77f23d389c71f\": rpc error: code = NotFound desc = could not find container \"a2843489ca9fdb173b509462c191f34b6d176397761ce07563d77f23d389c71f\": container with ID starting with a2843489ca9fdb173b509462c191f34b6d176397761ce07563d77f23d389c71f not found: ID does not exist"
Sep 14 13:26:34.886814 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.886793    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4"]
Sep 14 13:26:34.892605 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.892552    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-bc851d-5644dcddc7-tdfn4"]
Sep 14 13:26:34.904209 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.904188    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877"]
Sep 14 13:26:34.909726 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.909703    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-bc851d-7bb95dbff4-fz877"]
Sep 14 13:26:34.921482 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.921458    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9lrr\" (UniqueName: \"kubernetes.io/projected/2d3c87e8-36a9-4212-89b4-7e1ba1b64b93-kube-api-access-s9lrr\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:34.921482 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:34.921483    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddm7c\" (UniqueName: \"kubernetes.io/projected/b7b603c0-b71c-4631-b269-bbd00ca584fe-kube-api-access-ddm7c\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:36.056882 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:36.056847    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="141cb3ed-d01b-403f-b518-80b393199f8d" path="/var/lib/kubelet/pods/141cb3ed-d01b-403f-b518-80b393199f8d/volumes"
Sep 14 13:26:36.057358 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:36.057200    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d3c87e8-36a9-4212-89b4-7e1ba1b64b93" path="/var/lib/kubelet/pods/2d3c87e8-36a9-4212-89b4-7e1ba1b64b93/volumes"
Sep 14 13:26:36.057485 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:36.057471    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e0bf3bf-6203-4820-8b6f-9029792fefc7" path="/var/lib/kubelet/pods/4e0bf3bf-6203-4820-8b6f-9029792fefc7/volumes"
Sep 14 13:26:36.057840 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:36.057826    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7b603c0-b71c-4631-b269-bbd00ca584fe" path="/var/lib/kubelet/pods/b7b603c0-b71c-4631-b269-bbd00ca584fe/volumes"
Sep 14 13:26:41.482380 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.482318    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f" podUID="7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c" containerName="main" containerID="cri-o://455e8b9ccef9f1bb7017fc686d03033946116751b403479ad85f194a1fc3fae8" gracePeriod=45
Sep 14 13:26:41.491136 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.491108    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl" podUID="12b7d6dd-2596-4576-8fd1-bcd213ecf6e5" containerName="main" containerID="cri-o://a87e76043568306fc0d6734335987378fefaab3db743d13b77ed107c894d5dcb" gracePeriod=45
Sep 14 13:26:41.514134 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.514086    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx" podUID="678f934a-1451-4a80-95d1-13bb86c9e298" containerName="main" containerID="cri-o://f6c56698cbb77612b17c3671f40e5871661792ad53df694cb042050056301f0e" gracePeriod=45
Sep 14 13:26:41.531491 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.531464    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q" podUID="7634a83e-5864-499a-9769-f29de26d5ad5" containerName="main" containerID="cri-o://bf5c04b05230857f22f2a5898dc6f7aa85c619352bfcab233d042b38a1995d02" gracePeriod=45
Sep 14 13:26:41.848069 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.848046    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:26:41.851603 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.851536    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:26:41.855860 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.855841    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:26:41.859414 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.859391    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:26:41.888482 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.888452    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-home\") pod \"678f934a-1451-4a80-95d1-13bb86c9e298\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") "
Sep 14 13:26:41.888620 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.888510    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-tls-certs\") pod \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") "
Sep 14 13:26:41.888620 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.888546    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-tmp-dir\") pod \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") "
Sep 14 13:26:41.888620 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.888575    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-dshm\") pod \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") "
Sep 14 13:26:41.888620 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.888613    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-dshm\") pod \"678f934a-1451-4a80-95d1-13bb86c9e298\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") "
Sep 14 13:26:41.888872 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.888638    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-tls-certs\") pod \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") "
Sep 14 13:26:41.888872 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.888675    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-dshm\") pod \"7634a83e-5864-499a-9769-f29de26d5ad5\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") "
Sep 14 13:26:41.888872 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.888699    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-home\") pod \"7634a83e-5864-499a-9769-f29de26d5ad5\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") "
Sep 14 13:26:41.888872 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.888727    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-dshm\") pod \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") "
Sep 14 13:26:41.888872 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.888764    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-tmp-dir\") pod \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") "
Sep 14 13:26:41.888872 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.888798    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-model-cache\") pod \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") "
Sep 14 13:26:41.888872 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.888825    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-model-cache\") pod \"7634a83e-5864-499a-9769-f29de26d5ad5\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") "
Sep 14 13:26:41.888872 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.888692    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-home" (OuterVolumeSpecName: "home") pod "678f934a-1451-4a80-95d1-13bb86c9e298" (UID: "678f934a-1451-4a80-95d1-13bb86c9e298"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:26:41.888872 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.888815    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c" (UID: "7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:26:41.888872 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.888864    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44f89\" (UniqueName: \"kubernetes.io/projected/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-kube-api-access-44f89\") pod \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") "
Sep 14 13:26:41.889445 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.888914    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-model-cache\") pod \"678f934a-1451-4a80-95d1-13bb86c9e298\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") "
Sep 14 13:26:41.889445 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.888938    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-tmp-dir\") pod \"678f934a-1451-4a80-95d1-13bb86c9e298\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") "
Sep 14 13:26:41.889445 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.888961    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-home\") pod \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") "
Sep 14 13:26:41.889445 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.888987    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-model-cache\") pod \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\" (UID: \"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5\") "
Sep 14 13:26:41.889445 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.889016    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-tmp-dir\") pod \"7634a83e-5864-499a-9769-f29de26d5ad5\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") "
Sep 14 13:26:41.889445 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.889046    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27jgb\" (UniqueName: \"kubernetes.io/projected/678f934a-1451-4a80-95d1-13bb86c9e298-kube-api-access-27jgb\") pod \"678f934a-1451-4a80-95d1-13bb86c9e298\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") "
Sep 14 13:26:41.889445 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.889071    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-home\") pod \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") "
Sep 14 13:26:41.889445 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.889098    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/678f934a-1451-4a80-95d1-13bb86c9e298-tls-certs\") pod \"678f934a-1451-4a80-95d1-13bb86c9e298\" (UID: \"678f934a-1451-4a80-95d1-13bb86c9e298\") "
Sep 14 13:26:41.889445 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.889130    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7634a83e-5864-499a-9769-f29de26d5ad5-tls-certs\") pod \"7634a83e-5864-499a-9769-f29de26d5ad5\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") "
Sep 14 13:26:41.889445 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.889157    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7kfg\" (UniqueName: \"kubernetes.io/projected/7634a83e-5864-499a-9769-f29de26d5ad5-kube-api-access-c7kfg\") pod \"7634a83e-5864-499a-9769-f29de26d5ad5\" (UID: \"7634a83e-5864-499a-9769-f29de26d5ad5\") "
Sep 14 13:26:41.889445 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.889185    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zr8r4\" (UniqueName: \"kubernetes.io/projected/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-kube-api-access-zr8r4\") pod \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\" (UID: \"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c\") "
Sep 14 13:26:41.891893 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.891414    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-dshm" (OuterVolumeSpecName: "dshm") pod "7634a83e-5864-499a-9769-f29de26d5ad5" (UID: "7634a83e-5864-499a-9769-f29de26d5ad5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:26:41.891893 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.891668    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "12b7d6dd-2596-4576-8fd1-bcd213ecf6e5" (UID: "12b7d6dd-2596-4576-8fd1-bcd213ecf6e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:26:41.891893 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.891672    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c" (UID: "7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:26:41.891893 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.891816    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-dshm" (OuterVolumeSpecName: "dshm") pod "678f934a-1451-4a80-95d1-13bb86c9e298" (UID: "678f934a-1451-4a80-95d1-13bb86c9e298"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:26:41.892173 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.891922    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7634a83e-5864-499a-9769-f29de26d5ad5" (UID: "7634a83e-5864-499a-9769-f29de26d5ad5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:26:41.892235 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.892150    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-home" (OuterVolumeSpecName: "home") pod "12b7d6dd-2596-4576-8fd1-bcd213ecf6e5" (UID: "12b7d6dd-2596-4576-8fd1-bcd213ecf6e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:26:41.892295 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.892250    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-home" (OuterVolumeSpecName: "home") pod "7634a83e-5864-499a-9769-f29de26d5ad5" (UID: "7634a83e-5864-499a-9769-f29de26d5ad5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:26:41.893138 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.892456    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-model-cache" (OuterVolumeSpecName: "model-cache") pod "7634a83e-5864-499a-9769-f29de26d5ad5" (UID: "7634a83e-5864-499a-9769-f29de26d5ad5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:26:41.893138 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.892489    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12b7d6dd-2596-4576-8fd1-bcd213ecf6e5" (UID: "12b7d6dd-2596-4576-8fd1-bcd213ecf6e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:26:41.893138 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.892532    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-home\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.893138 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.892551    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-tmp-dir\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.893138 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.892565    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-dshm\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.893138 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.892578    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-tls-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.893138 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.892591    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-dshm\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.893138 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.892602    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-home\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.893138 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.892614    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-home\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.893138 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.892627    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-model-cache\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.893138 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.892639    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-tmp-dir\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.893138 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.892646    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12b7d6dd-2596-4576-8fd1-bcd213ecf6e5" (UID: "12b7d6dd-2596-4576-8fd1-bcd213ecf6e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:26:41.893138 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.892900    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-model-cache" (OuterVolumeSpecName: "model-cache") pod "678f934a-1451-4a80-95d1-13bb86c9e298" (UID: "678f934a-1451-4a80-95d1-13bb86c9e298"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:26:41.893138 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.892914    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c" (UID: "7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:26:41.893950 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.893197    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "678f934a-1451-4a80-95d1-13bb86c9e298" (UID: "678f934a-1451-4a80-95d1-13bb86c9e298"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:26:41.893950 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.893391    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-home" (OuterVolumeSpecName: "home") pod "7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c" (UID: "7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:26:41.894709 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.894656    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-dshm" (OuterVolumeSpecName: "dshm") pod "12b7d6dd-2596-4576-8fd1-bcd213ecf6e5" (UID: "12b7d6dd-2596-4576-8fd1-bcd213ecf6e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:26:41.895378 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.895352    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-kube-api-access-44f89" (OuterVolumeSpecName: "kube-api-access-44f89") pod "12b7d6dd-2596-4576-8fd1-bcd213ecf6e5" (UID: "12b7d6dd-2596-4576-8fd1-bcd213ecf6e5"). InnerVolumeSpecName "kube-api-access-44f89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:26:41.896126 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.896076    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-kube-api-access-zr8r4" (OuterVolumeSpecName: "kube-api-access-zr8r4") pod "7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c" (UID: "7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c"). InnerVolumeSpecName "kube-api-access-zr8r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:26:41.896560 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.896411    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7634a83e-5864-499a-9769-f29de26d5ad5-kube-api-access-c7kfg" (OuterVolumeSpecName: "kube-api-access-c7kfg") pod "7634a83e-5864-499a-9769-f29de26d5ad5" (UID: "7634a83e-5864-499a-9769-f29de26d5ad5"). InnerVolumeSpecName "kube-api-access-c7kfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:26:41.896560 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.896490    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-dshm" (OuterVolumeSpecName: "dshm") pod "7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c" (UID: "7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:26:41.896560 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.896535    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/678f934a-1451-4a80-95d1-13bb86c9e298-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "678f934a-1451-4a80-95d1-13bb86c9e298" (UID: "678f934a-1451-4a80-95d1-13bb86c9e298"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:26:41.897020 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.896977    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/678f934a-1451-4a80-95d1-13bb86c9e298-kube-api-access-27jgb" (OuterVolumeSpecName: "kube-api-access-27jgb") pod "678f934a-1451-4a80-95d1-13bb86c9e298" (UID: "678f934a-1451-4a80-95d1-13bb86c9e298"). InnerVolumeSpecName "kube-api-access-27jgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:26:41.897957 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.897935    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7634a83e-5864-499a-9769-f29de26d5ad5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7634a83e-5864-499a-9769-f29de26d5ad5" (UID: "7634a83e-5864-499a-9769-f29de26d5ad5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:26:41.906596 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.906572    2557 generic.go:356] "Generic (PLEG): container finished" podID="7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c" containerID="455e8b9ccef9f1bb7017fc686d03033946116751b403479ad85f194a1fc3fae8" exitCode=0
Sep 14 13:26:41.906705 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.906615    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"
Sep 14 13:26:41.906705 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.906667    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f" event={"ID":"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c","Type":"ContainerDied","Data":"455e8b9ccef9f1bb7017fc686d03033946116751b403479ad85f194a1fc3fae8"}
Sep 14 13:26:41.906827 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.906715    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f" event={"ID":"7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c","Type":"ContainerDied","Data":"14f755a0d131d66588ab6f275bc2cc65c0549a8b47af057a8417928dab160204"}
Sep 14 13:26:41.906827 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.906766    2557 scope.go:117] "RemoveContainer" containerID="455e8b9ccef9f1bb7017fc686d03033946116751b403479ad85f194a1fc3fae8"
Sep 14 13:26:41.908557 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.908321    2557 generic.go:356] "Generic (PLEG): container finished" podID="12b7d6dd-2596-4576-8fd1-bcd213ecf6e5" containerID="a87e76043568306fc0d6734335987378fefaab3db743d13b77ed107c894d5dcb" exitCode=0
Sep 14 13:26:41.908557 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.908381    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"
Sep 14 13:26:41.908557 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.908464    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl" event={"ID":"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5","Type":"ContainerDied","Data":"a87e76043568306fc0d6734335987378fefaab3db743d13b77ed107c894d5dcb"}
Sep 14 13:26:41.908557 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.908497    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl" event={"ID":"12b7d6dd-2596-4576-8fd1-bcd213ecf6e5","Type":"ContainerDied","Data":"3506e93a03265220f5c340fd41f5ef3cae0f4732b25a03b48f0e1d1cf3b5a5d9"}
Sep 14 13:26:41.910891 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.910599    2557 generic.go:356] "Generic (PLEG): container finished" podID="7634a83e-5864-499a-9769-f29de26d5ad5" containerID="bf5c04b05230857f22f2a5898dc6f7aa85c619352bfcab233d042b38a1995d02" exitCode=0
Sep 14 13:26:41.910891 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.910782    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q" event={"ID":"7634a83e-5864-499a-9769-f29de26d5ad5","Type":"ContainerDied","Data":"bf5c04b05230857f22f2a5898dc6f7aa85c619352bfcab233d042b38a1995d02"}
Sep 14 13:26:41.910891 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.910815    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q" event={"ID":"7634a83e-5864-499a-9769-f29de26d5ad5","Type":"ContainerDied","Data":"d928ac68919074c1423d2cdcba6402ef6fa61515367053344b3772117f0794f3"}
Sep 14 13:26:41.911167 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.910913    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"
Sep 14 13:26:41.915208 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.915185    2557 generic.go:356] "Generic (PLEG): container finished" podID="678f934a-1451-4a80-95d1-13bb86c9e298" containerID="f6c56698cbb77612b17c3671f40e5871661792ad53df694cb042050056301f0e" exitCode=0
Sep 14 13:26:41.915326 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.915247    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx" event={"ID":"678f934a-1451-4a80-95d1-13bb86c9e298","Type":"ContainerDied","Data":"f6c56698cbb77612b17c3671f40e5871661792ad53df694cb042050056301f0e"}
Sep 14 13:26:41.915326 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.915278    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx" event={"ID":"678f934a-1451-4a80-95d1-13bb86c9e298","Type":"ContainerDied","Data":"28b47b0a54efa0e24c3f8623a9f50ff70ccfb74f37512bd76cd4f869c7945c51"}
Sep 14 13:26:41.915440 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.915350    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"
Sep 14 13:26:41.924022 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.924002    2557 scope.go:117] "RemoveContainer" containerID="455e8b9ccef9f1bb7017fc686d03033946116751b403479ad85f194a1fc3fae8"
Sep 14 13:26:41.924428 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:26:41.924395    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"455e8b9ccef9f1bb7017fc686d03033946116751b403479ad85f194a1fc3fae8\": container with ID starting with 455e8b9ccef9f1bb7017fc686d03033946116751b403479ad85f194a1fc3fae8 not found: ID does not exist" containerID="455e8b9ccef9f1bb7017fc686d03033946116751b403479ad85f194a1fc3fae8"
Sep 14 13:26:41.924521 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.924439    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"455e8b9ccef9f1bb7017fc686d03033946116751b403479ad85f194a1fc3fae8"} err="failed to get container status \"455e8b9ccef9f1bb7017fc686d03033946116751b403479ad85f194a1fc3fae8\": rpc error: code = NotFound desc = could not find container \"455e8b9ccef9f1bb7017fc686d03033946116751b403479ad85f194a1fc3fae8\": container with ID starting with 455e8b9ccef9f1bb7017fc686d03033946116751b403479ad85f194a1fc3fae8 not found: ID does not exist"
Sep 14 13:26:41.924521 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.924478    2557 scope.go:117] "RemoveContainer" containerID="a87e76043568306fc0d6734335987378fefaab3db743d13b77ed107c894d5dcb"
Sep 14 13:26:41.937187 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.937161    2557 scope.go:117] "RemoveContainer" containerID="a87e76043568306fc0d6734335987378fefaab3db743d13b77ed107c894d5dcb"
Sep 14 13:26:41.937506 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:26:41.937483    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87e76043568306fc0d6734335987378fefaab3db743d13b77ed107c894d5dcb\": container with ID starting with a87e76043568306fc0d6734335987378fefaab3db743d13b77ed107c894d5dcb not found: ID does not exist" containerID="a87e76043568306fc0d6734335987378fefaab3db743d13b77ed107c894d5dcb"
Sep 14 13:26:41.937579 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.937516    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87e76043568306fc0d6734335987378fefaab3db743d13b77ed107c894d5dcb"} err="failed to get container status \"a87e76043568306fc0d6734335987378fefaab3db743d13b77ed107c894d5dcb\": rpc error: code = NotFound desc = could not find container \"a87e76043568306fc0d6734335987378fefaab3db743d13b77ed107c894d5dcb\": container with ID starting with a87e76043568306fc0d6734335987378fefaab3db743d13b77ed107c894d5dcb not found: ID does not exist"
Sep 14 13:26:41.937579 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.937538    2557 scope.go:117] "RemoveContainer" containerID="bf5c04b05230857f22f2a5898dc6f7aa85c619352bfcab233d042b38a1995d02"
Sep 14 13:26:41.947951 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.947924    2557 scope.go:117] "RemoveContainer" containerID="bf5c04b05230857f22f2a5898dc6f7aa85c619352bfcab233d042b38a1995d02"
Sep 14 13:26:41.948380 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:26:41.948245    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf5c04b05230857f22f2a5898dc6f7aa85c619352bfcab233d042b38a1995d02\": container with ID starting with bf5c04b05230857f22f2a5898dc6f7aa85c619352bfcab233d042b38a1995d02 not found: ID does not exist" containerID="bf5c04b05230857f22f2a5898dc6f7aa85c619352bfcab233d042b38a1995d02"
Sep 14 13:26:41.948380 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.948278    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf5c04b05230857f22f2a5898dc6f7aa85c619352bfcab233d042b38a1995d02"} err="failed to get container status \"bf5c04b05230857f22f2a5898dc6f7aa85c619352bfcab233d042b38a1995d02\": rpc error: code = NotFound desc = could not find container \"bf5c04b05230857f22f2a5898dc6f7aa85c619352bfcab233d042b38a1995d02\": container with ID starting with bf5c04b05230857f22f2a5898dc6f7aa85c619352bfcab233d042b38a1995d02 not found: ID does not exist"
Sep 14 13:26:41.948380 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.948299    2557 scope.go:117] "RemoveContainer" containerID="f6c56698cbb77612b17c3671f40e5871661792ad53df694cb042050056301f0e"
Sep 14 13:26:41.951859 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.951838    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"]
Sep 14 13:26:41.957114 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.957091    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b3d6c5/premium-w4-b3d6c5-kserve-6cb94675b7-8j44q"]
Sep 14 13:26:41.959454 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.959436    2557 scope.go:117] "RemoveContainer" containerID="f6c56698cbb77612b17c3671f40e5871661792ad53df694cb042050056301f0e"
Sep 14 13:26:41.959925 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:26:41.959887    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c56698cbb77612b17c3671f40e5871661792ad53df694cb042050056301f0e\": container with ID starting with f6c56698cbb77612b17c3671f40e5871661792ad53df694cb042050056301f0e not found: ID does not exist" containerID="f6c56698cbb77612b17c3671f40e5871661792ad53df694cb042050056301f0e"
Sep 14 13:26:41.960007 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.959928    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c56698cbb77612b17c3671f40e5871661792ad53df694cb042050056301f0e"} err="failed to get container status \"f6c56698cbb77612b17c3671f40e5871661792ad53df694cb042050056301f0e\": rpc error: code = NotFound desc = could not find container \"f6c56698cbb77612b17c3671f40e5871661792ad53df694cb042050056301f0e\": container with ID starting with f6c56698cbb77612b17c3671f40e5871661792ad53df694cb042050056301f0e not found: ID does not exist"
Sep 14 13:26:41.973019 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.972997    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"]
Sep 14 13:26:41.975245 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.975225    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-bc851d/embedding-w0-bc851d-kserve-769f4795c5-p5c7f"]
Sep 14 13:26:41.984011 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.983989    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"]
Sep 14 13:26:41.991518 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.991495    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-bc851d/premium-w0-bc851d-kserve-6d457757c9-2r9sl"]
Sep 14 13:26:41.993632 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.993598    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7634a83e-5864-499a-9769-f29de26d5ad5-model-cache\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.993716 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.993636    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44f89\" (UniqueName: \"kubernetes.io/projected/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-kube-api-access-44f89\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.993716 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.993652    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-model-cache\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.993716 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.993665    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/678f934a-1451-4a80-95d1-13bb86c9e298-tmp-dir\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.993716 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.993678    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27jgb\" (UniqueName: \"kubernetes.io/projected/678f934a-1451-4a80-95d1-13bb86c9e298-kube-api-access-27jgb\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.993716 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.993690    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-home\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.993716 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.993703    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/678f934a-1451-4a80-95d1-13bb86c9e298-tls-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.993716 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.993715    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7634a83e-5864-499a-9769-f29de26d5ad5-tls-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.994040 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.993727    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7kfg\" (UniqueName: \"kubernetes.io/projected/7634a83e-5864-499a-9769-f29de26d5ad5-kube-api-access-c7kfg\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.994040 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.993753    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr8r4\" (UniqueName: \"kubernetes.io/projected/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-kube-api-access-zr8r4\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.994040 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.993766    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-tls-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.994040 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.993778    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-dshm\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.994040 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.993789    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-dshm\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.994040 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.993801    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5-tmp-dir\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:41.994040 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:41.993812    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c-model-cache\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:42.002906 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:42.002843    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"]
Sep 14 13:26:42.008688 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:42.008663    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b3d6c5/distinct-w4-b3d6c5-kserve-8f6fd84fd-vcxpx"]
Sep 14 13:26:42.049509 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:42.049478    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12b7d6dd-2596-4576-8fd1-bcd213ecf6e5" path="/var/lib/kubelet/pods/12b7d6dd-2596-4576-8fd1-bcd213ecf6e5/volumes"
Sep 14 13:26:42.049990 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:42.049974    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="678f934a-1451-4a80-95d1-13bb86c9e298" path="/var/lib/kubelet/pods/678f934a-1451-4a80-95d1-13bb86c9e298/volumes"
Sep 14 13:26:42.050385 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:42.050370    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7634a83e-5864-499a-9769-f29de26d5ad5" path="/var/lib/kubelet/pods/7634a83e-5864-499a-9769-f29de26d5ad5/volumes"
Sep 14 13:26:42.050937 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:42.050919    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c" path="/var/lib/kubelet/pods/7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c/volumes"
Sep 14 13:26:44.070462 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.070428    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n"]
Sep 14 13:26:44.070944 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.070694    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n" podUID="876b1975-7474-4482-8551-da99ab4a1f26" containerName="payload-processing" containerID="cri-o://433dc05de1b425b9d224d69c66d3506e43abb54b37059124ec940f5a45bdd6e2" gracePeriod=30
Sep 14 13:26:44.076690 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.076665    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr"]
Sep 14 13:26:44.078521 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.076907    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr" podUID="fad7cd32-9384-45f1-bcbd-bcdb5ae40361" containerName="payload-pre-processing" containerID="cri-o://a985139d425b07f0d5442a735cca84d76b9890db9252b491a2e5152ddbda976a" gracePeriod=30
Sep 14 13:26:44.219296 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.219064    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3d6c5-qbwk5"]
Sep 14 13:26:44.222050 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.222018    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3d6c5-qbwk5"]
Sep 14 13:26:44.384484 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.384458    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n"
Sep 14 13:26:44.387211 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.387193    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr"
Sep 14 13:26:44.418396 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.418361    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzrp7\" (UniqueName: \"kubernetes.io/projected/fad7cd32-9384-45f1-bcbd-bcdb5ae40361-kube-api-access-kzrp7\") pod \"fad7cd32-9384-45f1-bcbd-bcdb5ae40361\" (UID: \"fad7cd32-9384-45f1-bcbd-bcdb5ae40361\") "
Sep 14 13:26:44.418396 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.418399    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-529nz\" (UniqueName: \"kubernetes.io/projected/876b1975-7474-4482-8551-da99ab4a1f26-kube-api-access-529nz\") pod \"876b1975-7474-4482-8551-da99ab4a1f26\" (UID: \"876b1975-7474-4482-8551-da99ab4a1f26\") "
Sep 14 13:26:44.418623 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.418499    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad7cd32-9384-45f1-bcbd-bcdb5ae40361-plugins-config-volume\") pod \"fad7cd32-9384-45f1-bcbd-bcdb5ae40361\" (UID: \"fad7cd32-9384-45f1-bcbd-bcdb5ae40361\") "
Sep 14 13:26:44.418795 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.418771    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/876b1975-7474-4482-8551-da99ab4a1f26-plugins-config-volume\") pod \"876b1975-7474-4482-8551-da99ab4a1f26\" (UID: \"876b1975-7474-4482-8551-da99ab4a1f26\") "
Sep 14 13:26:44.418795 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.418940    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad7cd32-9384-45f1-bcbd-bcdb5ae40361-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fad7cd32-9384-45f1-bcbd-bcdb5ae40361" (UID: "fad7cd32-9384-45f1-bcbd-bcdb5ae40361"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:26:44.418795 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.419252    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/876b1975-7474-4482-8551-da99ab4a1f26-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "876b1975-7474-4482-8551-da99ab4a1f26" (UID: "876b1975-7474-4482-8551-da99ab4a1f26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:26:44.418795 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.419296    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad7cd32-9384-45f1-bcbd-bcdb5ae40361-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:44.421931 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.421396    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/876b1975-7474-4482-8551-da99ab4a1f26-kube-api-access-529nz" (OuterVolumeSpecName: "kube-api-access-529nz") pod "876b1975-7474-4482-8551-da99ab4a1f26" (UID: "876b1975-7474-4482-8551-da99ab4a1f26"). InnerVolumeSpecName "kube-api-access-529nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:26:44.423168 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.422431    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad7cd32-9384-45f1-bcbd-bcdb5ae40361-kube-api-access-kzrp7" (OuterVolumeSpecName: "kube-api-access-kzrp7") pod "fad7cd32-9384-45f1-bcbd-bcdb5ae40361" (UID: "fad7cd32-9384-45f1-bcbd-bcdb5ae40361"). InnerVolumeSpecName "kube-api-access-kzrp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:26:44.519820 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.519780    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/876b1975-7474-4482-8551-da99ab4a1f26-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:44.519820 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.519807    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzrp7\" (UniqueName: \"kubernetes.io/projected/fad7cd32-9384-45f1-bcbd-bcdb5ae40361-kube-api-access-kzrp7\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:44.519820 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.519818    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-529nz\" (UniqueName: \"kubernetes.io/projected/876b1975-7474-4482-8551-da99ab4a1f26-kube-api-access-529nz\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:44.936154 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.936119    2557 generic.go:356] "Generic (PLEG): container finished" podID="fad7cd32-9384-45f1-bcbd-bcdb5ae40361" containerID="a985139d425b07f0d5442a735cca84d76b9890db9252b491a2e5152ddbda976a" exitCode=0
Sep 14 13:26:44.936372 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.936166    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr"
Sep 14 13:26:44.936372 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.936205    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr" event={"ID":"fad7cd32-9384-45f1-bcbd-bcdb5ae40361","Type":"ContainerDied","Data":"a985139d425b07f0d5442a735cca84d76b9890db9252b491a2e5152ddbda976a"}
Sep 14 13:26:44.936372 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.936316    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr" event={"ID":"fad7cd32-9384-45f1-bcbd-bcdb5ae40361","Type":"ContainerDied","Data":"e2f96e6e37621f377e6bd6ccbbaa121580ed135e9e7231e65d9d36eb0c483689"}
Sep 14 13:26:44.936372 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.936360    2557 scope.go:117] "RemoveContainer" containerID="a985139d425b07f0d5442a735cca84d76b9890db9252b491a2e5152ddbda976a"
Sep 14 13:26:44.937837 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.937810    2557 generic.go:356] "Generic (PLEG): container finished" podID="876b1975-7474-4482-8551-da99ab4a1f26" containerID="433dc05de1b425b9d224d69c66d3506e43abb54b37059124ec940f5a45bdd6e2" exitCode=0
Sep 14 13:26:44.937981 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.937838    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n" event={"ID":"876b1975-7474-4482-8551-da99ab4a1f26","Type":"ContainerDied","Data":"433dc05de1b425b9d224d69c66d3506e43abb54b37059124ec940f5a45bdd6e2"}
Sep 14 13:26:44.937981 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.937857    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n"
Sep 14 13:26:44.937981 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.937888    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n" event={"ID":"876b1975-7474-4482-8551-da99ab4a1f26","Type":"ContainerDied","Data":"eef0090433f9a99d70e5fdfd489c8933db43f71024a008212ea409ba76dcd105"}
Sep 14 13:26:44.946206 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.946186    2557 scope.go:117] "RemoveContainer" containerID="a985139d425b07f0d5442a735cca84d76b9890db9252b491a2e5152ddbda976a"
Sep 14 13:26:44.946535 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:26:44.946515    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a985139d425b07f0d5442a735cca84d76b9890db9252b491a2e5152ddbda976a\": container with ID starting with a985139d425b07f0d5442a735cca84d76b9890db9252b491a2e5152ddbda976a not found: ID does not exist" containerID="a985139d425b07f0d5442a735cca84d76b9890db9252b491a2e5152ddbda976a"
Sep 14 13:26:44.946596 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.946544    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a985139d425b07f0d5442a735cca84d76b9890db9252b491a2e5152ddbda976a"} err="failed to get container status \"a985139d425b07f0d5442a735cca84d76b9890db9252b491a2e5152ddbda976a\": rpc error: code = NotFound desc = could not find container \"a985139d425b07f0d5442a735cca84d76b9890db9252b491a2e5152ddbda976a\": container with ID starting with a985139d425b07f0d5442a735cca84d76b9890db9252b491a2e5152ddbda976a not found: ID does not exist"
Sep 14 13:26:44.946596 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.946561    2557 scope.go:117] "RemoveContainer" containerID="433dc05de1b425b9d224d69c66d3506e43abb54b37059124ec940f5a45bdd6e2"
Sep 14 13:26:44.955732 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.955701    2557 scope.go:117] "RemoveContainer" containerID="433dc05de1b425b9d224d69c66d3506e43abb54b37059124ec940f5a45bdd6e2"
Sep 14 13:26:44.956034 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:26:44.956013    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"433dc05de1b425b9d224d69c66d3506e43abb54b37059124ec940f5a45bdd6e2\": container with ID starting with 433dc05de1b425b9d224d69c66d3506e43abb54b37059124ec940f5a45bdd6e2 not found: ID does not exist" containerID="433dc05de1b425b9d224d69c66d3506e43abb54b37059124ec940f5a45bdd6e2"
Sep 14 13:26:44.956120 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.956039    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"433dc05de1b425b9d224d69c66d3506e43abb54b37059124ec940f5a45bdd6e2"} err="failed to get container status \"433dc05de1b425b9d224d69c66d3506e43abb54b37059124ec940f5a45bdd6e2\": rpc error: code = NotFound desc = could not find container \"433dc05de1b425b9d224d69c66d3506e43abb54b37059124ec940f5a45bdd6e2\": container with ID starting with 433dc05de1b425b9d224d69c66d3506e43abb54b37059124ec940f5a45bdd6e2 not found: ID does not exist"
Sep 14 13:26:44.962829 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.962809    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr"]
Sep 14 13:26:44.967985 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.967964    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b3d6c5-7945f5bf78-qk8kr"]
Sep 14 13:26:44.978794 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.978774    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n"]
Sep 14 13:26:44.982468 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:44.982445    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b3d6c5-678f6d565-vgm8n"]
Sep 14 13:26:46.049658 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:46.049629    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="876b1975-7474-4482-8551-da99ab4a1f26" path="/var/lib/kubelet/pods/876b1975-7474-4482-8551-da99ab4a1f26/volumes"
Sep 14 13:26:46.050074 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:46.050002    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d23896d-7a76-43cb-ab13-b49d79f23edc" path="/var/lib/kubelet/pods/9d23896d-7a76-43cb-ab13-b49d79f23edc/volumes"
Sep 14 13:26:46.050349 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:46.050331    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fad7cd32-9384-45f1-bcbd-bcdb5ae40361" path="/var/lib/kubelet/pods/fad7cd32-9384-45f1-bcbd-bcdb5ae40361/volumes"
Sep 14 13:26:52.830286 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:52.830256    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"]
Sep 14 13:26:52.830727 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:52.830493    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7" podUID="6836e788-4b82-44b3-bfb9-b9718848d0c0" containerName="istio-proxy" containerID="cri-o://4c8563a7806796c3bd765c69d83f208a73443d91d8a248c83521094c6739342f" gracePeriod=30
Sep 14 13:26:58.025823 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.025786    2557 generic.go:356] "Generic (PLEG): container finished" podID="6836e788-4b82-44b3-bfb9-b9718848d0c0" containerID="4c8563a7806796c3bd765c69d83f208a73443d91d8a248c83521094c6739342f" exitCode=0
Sep 14 13:26:58.026212 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.025853    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7" event={"ID":"6836e788-4b82-44b3-bfb9-b9718848d0c0","Type":"ContainerDied","Data":"4c8563a7806796c3bd765c69d83f208a73443d91d8a248c83521094c6739342f"}
Sep 14 13:26:58.026212 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.025919    2557 scope.go:117] "RemoveContainer" containerID="201ac78560540a8eb5cc24e8764b85d749e538d6453dc9566027aa627091071b"
Sep 14 13:26:58.088251 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.088227    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:26:58.248331 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.248300    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6836e788-4b82-44b3-bfb9-b9718848d0c0-istiod-ca-cert\") pod \"6836e788-4b82-44b3-bfb9-b9718848d0c0\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") "
Sep 14 13:26:58.248505 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.248340    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-credential-socket\") pod \"6836e788-4b82-44b3-bfb9-b9718848d0c0\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") "
Sep 14 13:26:58.248505 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.248366    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-podinfo\") pod \"6836e788-4b82-44b3-bfb9-b9718848d0c0\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") "
Sep 14 13:26:58.248505 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.248391    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-data\") pod \"6836e788-4b82-44b3-bfb9-b9718848d0c0\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") "
Sep 14 13:26:58.248505 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.248410    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzpb9\" (UniqueName: \"kubernetes.io/projected/6836e788-4b82-44b3-bfb9-b9718848d0c0-kube-api-access-mzpb9\") pod \"6836e788-4b82-44b3-bfb9-b9718848d0c0\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") "
Sep 14 13:26:58.248505 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.248503    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-workload-socket\") pod \"6836e788-4b82-44b3-bfb9-b9718848d0c0\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") "
Sep 14 13:26:58.248795 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.248548    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-envoy\") pod \"6836e788-4b82-44b3-bfb9-b9718848d0c0\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") "
Sep 14 13:26:58.248795 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.248597    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-workload-certs\") pod \"6836e788-4b82-44b3-bfb9-b9718848d0c0\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") "
Sep 14 13:26:58.248899 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.248809    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-data" (OuterVolumeSpecName: "istio-data") pod "6836e788-4b82-44b3-bfb9-b9718848d0c0" (UID: "6836e788-4b82-44b3-bfb9-b9718848d0c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:26:58.248899 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.248839    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6836e788-4b82-44b3-bfb9-b9718848d0c0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6836e788-4b82-44b3-bfb9-b9718848d0c0" (UID: "6836e788-4b82-44b3-bfb9-b9718848d0c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:26:58.248899 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.248850    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6836e788-4b82-44b3-bfb9-b9718848d0c0" (UID: "6836e788-4b82-44b3-bfb9-b9718848d0c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:26:58.248899 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.248868    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-token\") pod \"6836e788-4b82-44b3-bfb9-b9718848d0c0\" (UID: \"6836e788-4b82-44b3-bfb9-b9718848d0c0\") "
Sep 14 13:26:58.249104 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.248985    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6836e788-4b82-44b3-bfb9-b9718848d0c0" (UID: "6836e788-4b82-44b3-bfb9-b9718848d0c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:26:58.249182 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.249154    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6836e788-4b82-44b3-bfb9-b9718848d0c0" (UID: "6836e788-4b82-44b3-bfb9-b9718848d0c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:26:58.249266 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.249243    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-workload-socket\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:58.249324 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.249272    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-workload-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:58.249324 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.249285    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6836e788-4b82-44b3-bfb9-b9718848d0c0-istiod-ca-cert\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:58.249324 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.249297    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-credential-socket\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:58.249324 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.249309    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-data\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:58.251074 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.251042    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6836e788-4b82-44b3-bfb9-b9718848d0c0" (UID: "6836e788-4b82-44b3-bfb9-b9718848d0c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 13:26:58.251186 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.251125    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6836e788-4b82-44b3-bfb9-b9718848d0c0-kube-api-access-mzpb9" (OuterVolumeSpecName: "kube-api-access-mzpb9") pod "6836e788-4b82-44b3-bfb9-b9718848d0c0" (UID: "6836e788-4b82-44b3-bfb9-b9718848d0c0"). InnerVolumeSpecName "kube-api-access-mzpb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:26:58.251371 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.251343    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6836e788-4b82-44b3-bfb9-b9718848d0c0" (UID: "6836e788-4b82-44b3-bfb9-b9718848d0c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:26:58.251371 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.251348    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-token" (OuterVolumeSpecName: "istio-token") pod "6836e788-4b82-44b3-bfb9-b9718848d0c0" (UID: "6836e788-4b82-44b3-bfb9-b9718848d0c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:26:58.349768 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.349725    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-envoy\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:58.349768 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.349766    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-token\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:58.349950 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.349780    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6836e788-4b82-44b3-bfb9-b9718848d0c0-istio-podinfo\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:58.349950 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:58.349791    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzpb9\" (UniqueName: \"kubernetes.io/projected/6836e788-4b82-44b3-bfb9-b9718848d0c0-kube-api-access-mzpb9\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:26:59.032860 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:59.032827    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"
Sep 14 13:26:59.033305 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:59.032860    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7" event={"ID":"6836e788-4b82-44b3-bfb9-b9718848d0c0","Type":"ContainerDied","Data":"2d4a329524dd98fbccc599ee3ada51f4a2eaf8f5bbbab58de5b3502e60f8b7c1"}
Sep 14 13:26:59.033305 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:59.032918    2557 scope.go:117] "RemoveContainer" containerID="4c8563a7806796c3bd765c69d83f208a73443d91d8a248c83521094c6739342f"
Sep 14 13:26:59.055092 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:59.055067    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"]
Sep 14 13:26:59.058863 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:26:59.058841    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b3d6c5-openshift-default-6b77695f78-hnlm7"]
Sep 14 13:27:00.048286 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:00.048243    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6836e788-4b82-44b3-bfb9-b9718848d0c0" path="/var/lib/kubelet/pods/6836e788-4b82-44b3-bfb9-b9718848d0c0/volumes"
Sep 14 13:27:02.458606 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:02.458574    2557 scope.go:117] "RemoveContainer" containerID="cb7de97a703562397238af32f8f76ce1429d2a65cf5921f9f4edf6f31901f127"
Sep 14 13:27:10.206559 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.206492    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"]
Sep 14 13:27:10.207146 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207126    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d3c87e8-36a9-4212-89b4-7e1ba1b64b93" containerName="payload-processing"
Sep 14 13:27:10.207193 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207149    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d3c87e8-36a9-4212-89b4-7e1ba1b64b93" containerName="payload-processing"
Sep 14 13:27:10.207193 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207166    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c" containerName="main"
Sep 14 13:27:10.207193 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207175    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c" containerName="main"
Sep 14 13:27:10.207287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207192    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="876b1975-7474-4482-8551-da99ab4a1f26" containerName="payload-processing"
Sep 14 13:27:10.207287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207201    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="876b1975-7474-4482-8551-da99ab4a1f26" containerName="payload-processing"
Sep 14 13:27:10.207287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207209    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6836e788-4b82-44b3-bfb9-b9718848d0c0" containerName="istio-proxy"
Sep 14 13:27:10.207287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207218    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6836e788-4b82-44b3-bfb9-b9718848d0c0" containerName="istio-proxy"
Sep 14 13:27:10.207287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207240    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12b7d6dd-2596-4576-8fd1-bcd213ecf6e5" containerName="main"
Sep 14 13:27:10.207287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207249    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="12b7d6dd-2596-4576-8fd1-bcd213ecf6e5" containerName="main"
Sep 14 13:27:10.207287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207259    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6836e788-4b82-44b3-bfb9-b9718848d0c0" containerName="istio-proxy"
Sep 14 13:27:10.207287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207267    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6836e788-4b82-44b3-bfb9-b9718848d0c0" containerName="istio-proxy"
Sep 14 13:27:10.207287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207278    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e0bf3bf-6203-4820-8b6f-9029792fefc7" containerName="revoke-keys"
Sep 14 13:27:10.207287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207286    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e0bf3bf-6203-4820-8b6f-9029792fefc7" containerName="revoke-keys"
Sep 14 13:27:10.207608 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207296    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fad7cd32-9384-45f1-bcbd-bcdb5ae40361" containerName="payload-pre-processing"
Sep 14 13:27:10.207608 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207303    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fad7cd32-9384-45f1-bcbd-bcdb5ae40361" containerName="payload-pre-processing"
Sep 14 13:27:10.207608 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207314    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7634a83e-5864-499a-9769-f29de26d5ad5" containerName="main"
Sep 14 13:27:10.207608 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207322    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7634a83e-5864-499a-9769-f29de26d5ad5" containerName="main"
Sep 14 13:27:10.207608 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207331    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d23896d-7a76-43cb-ab13-b49d79f23edc" containerName="revoke-keys"
Sep 14 13:27:10.207608 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207338    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d23896d-7a76-43cb-ab13-b49d79f23edc" containerName="revoke-keys"
Sep 14 13:27:10.207608 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207348    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="678f934a-1451-4a80-95d1-13bb86c9e298" containerName="main"
Sep 14 13:27:10.207608 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207356    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="678f934a-1451-4a80-95d1-13bb86c9e298" containerName="main"
Sep 14 13:27:10.207608 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207366    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7b603c0-b71c-4631-b269-bbd00ca584fe" containerName="payload-pre-processing"
Sep 14 13:27:10.207608 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207374    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7b603c0-b71c-4631-b269-bbd00ca584fe" containerName="payload-pre-processing"
Sep 14 13:27:10.207608 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207457    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7b603c0-b71c-4631-b269-bbd00ca584fe" containerName="payload-pre-processing"
Sep 14 13:27:10.207608 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207469    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="678f934a-1451-4a80-95d1-13bb86c9e298" containerName="main"
Sep 14 13:27:10.207608 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207478    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d3c87e8-36a9-4212-89b4-7e1ba1b64b93" containerName="payload-processing"
Sep 14 13:27:10.207608 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207499    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7634a83e-5864-499a-9769-f29de26d5ad5" containerName="main"
Sep 14 13:27:10.207608 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207509    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="876b1975-7474-4482-8551-da99ab4a1f26" containerName="payload-processing"
Sep 14 13:27:10.207608 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207517    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d23896d-7a76-43cb-ab13-b49d79f23edc" containerName="revoke-keys"
Sep 14 13:27:10.207608 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207526    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f52a7f4-8cef-44d2-8f72-d8cf9c62a35c" containerName="main"
Sep 14 13:27:10.207608 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207533    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6836e788-4b82-44b3-bfb9-b9718848d0c0" containerName="istio-proxy"
Sep 14 13:27:10.207608 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207540    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e0bf3bf-6203-4820-8b6f-9029792fefc7" containerName="revoke-keys"
Sep 14 13:27:10.207608 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207549    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="fad7cd32-9384-45f1-bcbd-bcdb5ae40361" containerName="payload-pre-processing"
Sep 14 13:27:10.207608 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.207558    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="12b7d6dd-2596-4576-8fd1-bcd213ecf6e5" containerName="main"
Sep 14 13:27:10.211142 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.211126    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.214346 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.214327    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-622b87-openshift-default-dockercfg-4lw67\""
Sep 14 13:27:10.223078 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.223058    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"]
Sep 14 13:27:10.359030 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.358997    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-workload-certs\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.359200 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.359038    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed06c396-3158-4e8b-a961-89e905914221-istio-token\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.359200 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.359059    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4qts\" (UniqueName: \"kubernetes.io/projected/ed06c396-3158-4e8b-a961-89e905914221-kube-api-access-f4qts\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.359200 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.359080    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-credential-socket\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.359200 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.359148    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed06c396-3158-4e8b-a961-89e905914221-istiod-ca-cert\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.359200 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.359200    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed06c396-3158-4e8b-a961-89e905914221-istio-podinfo\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.359372 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.359225    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-istio-envoy\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.359372 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.359260    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-istio-data\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.359372 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.359309    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-workload-socket\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.459798 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.459708    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-istio-data\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.459798 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.459764    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-workload-socket\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.459798 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.459794    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-workload-certs\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.460134 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.459817    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed06c396-3158-4e8b-a961-89e905914221-istio-token\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.460134 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.459833    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4qts\" (UniqueName: \"kubernetes.io/projected/ed06c396-3158-4e8b-a961-89e905914221-kube-api-access-f4qts\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.460134 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.459853    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-credential-socket\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.460134 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.459874    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed06c396-3158-4e8b-a961-89e905914221-istiod-ca-cert\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.460134 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.459909    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed06c396-3158-4e8b-a961-89e905914221-istio-podinfo\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.460134 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.459942    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-istio-envoy\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.460409 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.460150    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-istio-data\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.460409 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.460164    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-workload-certs\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.460409 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.460232    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-workload-socket\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.460590 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.460566    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-credential-socket\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.460666 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.460626    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed06c396-3158-4e8b-a961-89e905914221-istiod-ca-cert\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.462164 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.462140    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-istio-envoy\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.462392 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.462375    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed06c396-3158-4e8b-a961-89e905914221-istio-podinfo\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.468088 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.468062    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed06c396-3158-4e8b-a961-89e905914221-istio-token\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.468088 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.468075    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4qts\" (UniqueName: \"kubernetes.io/projected/ed06c396-3158-4e8b-a961-89e905914221-kube-api-access-f4qts\") pod \"e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") " pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.522105 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.522078    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:10.806507 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.805912    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"]
Sep 14 13:27:10.825807 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.825770    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:27:10.826388 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.826365    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:27:10.826561 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:10.826543    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:27:11.103231 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:11.103148    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd" event={"ID":"ed06c396-3158-4e8b-a961-89e905914221","Type":"ContainerStarted","Data":"bf12b2d146f57f01df443b7c28545ebce09acccc8220219ab45ace15840de147"}
Sep 14 13:27:11.103231 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:11.103208    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd" event={"ID":"ed06c396-3158-4e8b-a961-89e905914221","Type":"ContainerStarted","Data":"b3e71e27beeda0d305b2821cb8283686dec38b241f86eb6b1a34501653edb5f5"}
Sep 14 13:27:11.132302 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:11.132254    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd" podStartSLOduration=1.132237805 podStartE2EDuration="1.132237805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:27:11.130975726 +0000 UTC m=+1749.565651246" watchObservedRunningTime="2026-09-14 13:27:11.132237805 +0000 UTC m=+1749.566913326"
Sep 14 13:27:11.522415 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:11.522386    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:11.527555 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:11.527531    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:12.111591 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:12.111558    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:12.114813 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:12.114786    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:17.759050 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:17.758997    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8"]
Sep 14 13:27:17.759714 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:17.759694    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6836e788-4b82-44b3-bfb9-b9718848d0c0" containerName="istio-proxy"
Sep 14 13:27:17.763107 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:17.763091    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8"
Sep 14 13:27:17.766335 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:17.766317    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-622b87\""
Sep 14 13:27:17.773405 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:17.773383    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8"]
Sep 14 13:27:17.824427 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:17.824395    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dcf5d12-54dd-4d49-862c-96c5a874bc74-plugins-config-volume\") pod \"payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8\" (UID: \"4dcf5d12-54dd-4d49-862c-96c5a874bc74\") " pod="openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8"
Sep 14 13:27:17.824561 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:17.824444    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qhtt\" (UniqueName: \"kubernetes.io/projected/4dcf5d12-54dd-4d49-862c-96c5a874bc74-kube-api-access-7qhtt\") pod \"payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8\" (UID: \"4dcf5d12-54dd-4d49-862c-96c5a874bc74\") " pod="openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8"
Sep 14 13:27:17.925618 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:17.925579    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dcf5d12-54dd-4d49-862c-96c5a874bc74-plugins-config-volume\") pod \"payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8\" (UID: \"4dcf5d12-54dd-4d49-862c-96c5a874bc74\") " pod="openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8"
Sep 14 13:27:17.925781 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:17.925634    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qhtt\" (UniqueName: \"kubernetes.io/projected/4dcf5d12-54dd-4d49-862c-96c5a874bc74-kube-api-access-7qhtt\") pod \"payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8\" (UID: \"4dcf5d12-54dd-4d49-862c-96c5a874bc74\") " pod="openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8"
Sep 14 13:27:17.926290 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:17.926271    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dcf5d12-54dd-4d49-862c-96c5a874bc74-plugins-config-volume\") pod \"payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8\" (UID: \"4dcf5d12-54dd-4d49-862c-96c5a874bc74\") " pod="openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8"
Sep 14 13:27:17.934522 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:17.934484    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qhtt\" (UniqueName: \"kubernetes.io/projected/4dcf5d12-54dd-4d49-862c-96c5a874bc74-kube-api-access-7qhtt\") pod \"payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8\" (UID: \"4dcf5d12-54dd-4d49-862c-96c5a874bc74\") " pod="openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8"
Sep 14 13:27:18.074214 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:18.074131    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8"
Sep 14 13:27:18.349694 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:18.349666    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8"]
Sep 14 13:27:18.351498 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:27:18.351473    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dcf5d12_54dd_4d49_862c_96c5a874bc74.slice/crio-61f387d94c1448715b846d2b0c705f7197ec23d7375bd70ee89a468135834bff WatchSource:0}: Error finding container 61f387d94c1448715b846d2b0c705f7197ec23d7375bd70ee89a468135834bff: Status 404 returned error can't find the container with id 61f387d94c1448715b846d2b0c705f7197ec23d7375bd70ee89a468135834bff
Sep 14 13:27:19.152167 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:19.152130    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8" event={"ID":"4dcf5d12-54dd-4d49-862c-96c5a874bc74","Type":"ContainerStarted","Data":"5b841d88d7560268fcccf585b9017bc3c65126dd1c9afa03b0c6c19b5a9c4428"}
Sep 14 13:27:19.152576 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:19.152192    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8" event={"ID":"4dcf5d12-54dd-4d49-862c-96c5a874bc74","Type":"ContainerStarted","Data":"61f387d94c1448715b846d2b0c705f7197ec23d7375bd70ee89a468135834bff"}
Sep 14 13:27:19.152576 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:19.152239    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8"
Sep 14 13:27:19.173268 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:19.173219    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8" podStartSLOduration=2.173202835 podStartE2EDuration="2.173202835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:27:19.169512444 +0000 UTC m=+1757.604187963" watchObservedRunningTime="2026-09-14 13:27:19.173202835 +0000 UTC m=+1757.607878354"
Sep 14 13:27:30.159539 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:30.159506    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8"
Sep 14 13:27:32.220270 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:27:32.220239    2557 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@8-22384-0.service\": RecentStats: unable to find data in memory cache]"
Sep 14 13:27:33.229044 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:33.229012    2557 generic.go:356] "Generic (PLEG): container finished" podID="ed06c396-3158-4e8b-a961-89e905914221" containerID="bf12b2d146f57f01df443b7c28545ebce09acccc8220219ab45ace15840de147" exitCode=0
Sep 14 13:27:33.229444 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:33.229087    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd" event={"ID":"ed06c396-3158-4e8b-a961-89e905914221","Type":"ContainerDied","Data":"bf12b2d146f57f01df443b7c28545ebce09acccc8220219ab45ace15840de147"}
Sep 14 13:27:33.229506 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:33.229489    2557 scope.go:117] "RemoveContainer" containerID="bf12b2d146f57f01df443b7c28545ebce09acccc8220219ab45ace15840de147"
Sep 14 13:27:33.230552 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:33.230513    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:27:33.230659 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:33.230611    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:27:33.230723 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:33.230668    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:27:34.236378 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:34.236334    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd" event={"ID":"ed06c396-3158-4e8b-a961-89e905914221","Type":"ContainerStarted","Data":"c790e9ce48f1d369539bd445dc5f9726e53d53f82af82e9264eb1e174b18d4d2"}
Sep 14 13:27:34.522714 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:34.522613    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:34.522947 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:34.522923    2557 patch_prober.go:28] interesting pod/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Sep 14 13:27:34.522999 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:34.522978    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd" podUID="ed06c396-3158-4e8b-a961-89e905914221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Sep 14 13:27:35.523123 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:35.523088    2557 patch_prober.go:28] interesting pod/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Sep 14 13:27:35.523497 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:35.523143    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd" podUID="ed06c396-3158-4e8b-a961-89e905914221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Sep 14 13:27:36.523003 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:36.522963    2557 patch_prober.go:28] interesting pod/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Sep 14 13:27:36.523185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:36.523028    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd" podUID="ed06c396-3158-4e8b-a961-89e905914221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Sep 14 13:27:37.522933 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:37.522895    2557 patch_prober.go:28] interesting pod/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Sep 14 13:27:37.523108 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:37.522965    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd" podUID="ed06c396-3158-4e8b-a961-89e905914221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Sep 14 13:27:38.522939 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:38.522894    2557 patch_prober.go:28] interesting pod/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Sep 14 13:27:38.523423 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:38.522956    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd" podUID="ed06c396-3158-4e8b-a961-89e905914221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Sep 14 13:27:39.600626 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:39.600593    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:39.601218 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:39.601195    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:39.602010 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:39.601990    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:27:43.831561 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:43.831531    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"]
Sep 14 13:27:43.835279 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:43.835257    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:43.837984 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:43.837962    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1cf25e-openshift-default-dockercfg-gfk4x\""
Sep 14 13:27:43.850647 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:43.850627    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"]
Sep 14 13:27:43.965294 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:43.965259    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-credential-socket\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:43.965294 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:43.965295    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-envoy\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:43.965520 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:43.965330    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzjcd\" (UniqueName: \"kubernetes.io/projected/2e7b8e99-ce68-4bac-9a90-10dbfc580209-kube-api-access-fzjcd\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:43.965520 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:43.965347    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-data\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:43.965520 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:43.965398    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-podinfo\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:43.965520 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:43.965413    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istiod-ca-cert\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:43.965520 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:43.965448    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-workload-certs\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:43.965520 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:43.965470    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-token\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:43.965772 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:43.965557    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-workload-socket\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:44.066823 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.066794    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-podinfo\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:44.066823 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.066827    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istiod-ca-cert\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:44.067063 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.067020    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-workload-certs\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:44.067129 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.067064    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-token\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:44.067129 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.067112    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-workload-socket\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:44.067238 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.067166    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-credential-socket\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:44.067238 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.067192    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-envoy\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:44.067352 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.067331    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzjcd\" (UniqueName: \"kubernetes.io/projected/2e7b8e99-ce68-4bac-9a90-10dbfc580209-kube-api-access-fzjcd\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:44.067414 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.067360    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-data\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:44.067470 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.067408    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-workload-certs\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:44.067521 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.067465    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-workload-socket\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:44.067521 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.067496    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-credential-socket\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:44.067605 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.067573    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istiod-ca-cert\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:44.067724 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.067709    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-data\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:44.069352 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.069329    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-podinfo\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:44.069465 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.069447    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-envoy\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:44.075053 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.075033    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-token\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:44.075250 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.075227    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzjcd\" (UniqueName: \"kubernetes.io/projected/2e7b8e99-ce68-4bac-9a90-10dbfc580209-kube-api-access-fzjcd\") pod \"e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") " pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:44.145530 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.145467    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:44.421561 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.421531    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"]
Sep 14 13:27:44.423119 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:27:44.423093    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7b8e99_ce68_4bac_9a90_10dbfc580209.slice/crio-193345ab3405120fd10d1ca57c754764155ee09bcb1ff87d004d7ca1bac89563 WatchSource:0}: Error finding container 193345ab3405120fd10d1ca57c754764155ee09bcb1ff87d004d7ca1bac89563: Status 404 returned error can't find the container with id 193345ab3405120fd10d1ca57c754764155ee09bcb1ff87d004d7ca1bac89563
Sep 14 13:27:44.425790 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.425733    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:27:44.425869 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.425838    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:27:44.425907 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:44.425869    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:27:45.304974 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:45.304942    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8" event={"ID":"2e7b8e99-ce68-4bac-9a90-10dbfc580209","Type":"ContainerStarted","Data":"7aff163bab6c22e5ebcff9a154cb9fe9a7c5fde683df2ca27a6a2a14a3db15b2"}
Sep 14 13:27:45.305350 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:45.304996    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8" event={"ID":"2e7b8e99-ce68-4bac-9a90-10dbfc580209","Type":"ContainerStarted","Data":"193345ab3405120fd10d1ca57c754764155ee09bcb1ff87d004d7ca1bac89563"}
Sep 14 13:27:45.330171 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:45.330121    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8" podStartSLOduration=2.33010475 podStartE2EDuration="2.33010475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:27:45.327769812 +0000 UTC m=+1783.762445330" watchObservedRunningTime="2026-09-14 13:27:45.33010475 +0000 UTC m=+1783.764780268"
Sep 14 13:27:46.145973 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:46.145937    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:46.152185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:46.152159    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:46.310883 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:46.310851    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:46.311857 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:46.311838    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:27:51.687562 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:51.687527    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j"]
Sep 14 13:27:51.694376 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:51.694161    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j"
Sep 14 13:27:51.696942 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:51.696915    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1cf25e\""
Sep 14 13:27:51.697143 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:51.697000    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-1cf25e-dockercfg-mdzkj\""
Sep 14 13:27:51.701929 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:51.701899    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j"]
Sep 14 13:27:51.844490 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:51.844452    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4fd4938-5f6d-49b7-9909-a3bad54689b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j\" (UID: \"e4fd4938-5f6d-49b7-9909-a3bad54689b4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j"
Sep 14 13:27:51.844704 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:51.844513    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvzld\" (UniqueName: \"kubernetes.io/projected/e4fd4938-5f6d-49b7-9909-a3bad54689b4-kube-api-access-fvzld\") pod \"payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j\" (UID: \"e4fd4938-5f6d-49b7-9909-a3bad54689b4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j"
Sep 14 13:27:51.945257 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:51.945222    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4fd4938-5f6d-49b7-9909-a3bad54689b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j\" (UID: \"e4fd4938-5f6d-49b7-9909-a3bad54689b4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j"
Sep 14 13:27:51.945437 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:51.945369    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvzld\" (UniqueName: \"kubernetes.io/projected/e4fd4938-5f6d-49b7-9909-a3bad54689b4-kube-api-access-fvzld\") pod \"payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j\" (UID: \"e4fd4938-5f6d-49b7-9909-a3bad54689b4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j"
Sep 14 13:27:51.945950 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:51.945930    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4fd4938-5f6d-49b7-9909-a3bad54689b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j\" (UID: \"e4fd4938-5f6d-49b7-9909-a3bad54689b4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j"
Sep 14 13:27:51.953992 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:51.953972    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvzld\" (UniqueName: \"kubernetes.io/projected/e4fd4938-5f6d-49b7-9909-a3bad54689b4-kube-api-access-fvzld\") pod \"payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j\" (UID: \"e4fd4938-5f6d-49b7-9909-a3bad54689b4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j"
Sep 14 13:27:52.007693 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:52.007664    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j"
Sep 14 13:27:52.279291 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:52.279216    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j"]
Sep 14 13:27:52.282539 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:27:52.282509    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4fd4938_5f6d_49b7_9909_a3bad54689b4.slice/crio-fce0f2337c572180b787622edbc780c34bbefb2a97703f074b60ff7a4c4b18b1 WatchSource:0}: Error finding container fce0f2337c572180b787622edbc780c34bbefb2a97703f074b60ff7a4c4b18b1: Status 404 returned error can't find the container with id fce0f2337c572180b787622edbc780c34bbefb2a97703f074b60ff7a4c4b18b1
Sep 14 13:27:52.341631 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:52.341605    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j" event={"ID":"e4fd4938-5f6d-49b7-9909-a3bad54689b4","Type":"ContainerStarted","Data":"fce0f2337c572180b787622edbc780c34bbefb2a97703f074b60ff7a4c4b18b1"}
Sep 14 13:27:53.348516 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:53.348471    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j" event={"ID":"e4fd4938-5f6d-49b7-9909-a3bad54689b4","Type":"ContainerStarted","Data":"2b8c4ca64b46d1d755adf3f108d466b3ac373179b4c4aab4ac93a5f03f91e79a"}
Sep 14 13:27:53.349284 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:53.348558    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j"
Sep 14 13:27:53.373208 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:27:53.373147    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j" podStartSLOduration=2.37313062 podStartE2EDuration="2.37313062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:27:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:27:53.371409531 +0000 UTC m=+1791.806085052" watchObservedRunningTime="2026-09-14 13:27:53.37313062 +0000 UTC m=+1791.807806183"
Sep 14 13:28:02.174734 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:02.174709    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwqnf_62d953d6-3b86-4c95-9a47-9579d88993a7/ovn-acl-logging/0.log"
Sep 14 13:28:02.175219 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:02.175076    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwqnf_62d953d6-3b86-4c95-9a47-9579d88993a7/ovn-acl-logging/0.log"
Sep 14 13:28:04.356339 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:04.356302    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j"
Sep 14 13:28:05.419832 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:05.419803    2557 generic.go:356] "Generic (PLEG): container finished" podID="2e7b8e99-ce68-4bac-9a90-10dbfc580209" containerID="7aff163bab6c22e5ebcff9a154cb9fe9a7c5fde683df2ca27a6a2a14a3db15b2" exitCode=0
Sep 14 13:28:05.420231 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:05.419879    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8" event={"ID":"2e7b8e99-ce68-4bac-9a90-10dbfc580209","Type":"ContainerDied","Data":"7aff163bab6c22e5ebcff9a154cb9fe9a7c5fde683df2ca27a6a2a14a3db15b2"}
Sep 14 13:28:05.420306 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:05.420291    2557 scope.go:117] "RemoveContainer" containerID="7aff163bab6c22e5ebcff9a154cb9fe9a7c5fde683df2ca27a6a2a14a3db15b2"
Sep 14 13:28:05.421483 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:05.421457    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:28:05.421536 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:05.421524    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:28:05.421574 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:05.421557    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:28:06.427197 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:06.427165    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8" event={"ID":"2e7b8e99-ce68-4bac-9a90-10dbfc580209","Type":"ContainerStarted","Data":"da66ecc96d8ea09c3fc8d05cbfab47f66b563ae5bb13499a8f76909f4b615d7a"}
Sep 14 13:28:07.145868 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:07.145837    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:28:07.146074 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:07.146052    2557 patch_prober.go:28] interesting pod/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Sep 14 13:28:07.146121 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:07.146103    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8" podUID="2e7b8e99-ce68-4bac-9a90-10dbfc580209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Sep 14 13:28:08.145907 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:08.145868    2557 patch_prober.go:28] interesting pod/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Sep 14 13:28:08.146327 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:08.145941    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8" podUID="2e7b8e99-ce68-4bac-9a90-10dbfc580209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Sep 14 13:28:09.146217 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:09.146184    2557 patch_prober.go:28] interesting pod/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Sep 14 13:28:09.146619 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:09.146240    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8" podUID="2e7b8e99-ce68-4bac-9a90-10dbfc580209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Sep 14 13:28:10.146672 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:10.146633    2557 patch_prober.go:28] interesting pod/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Sep 14 13:28:10.147226 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:10.146701    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8" podUID="2e7b8e99-ce68-4bac-9a90-10dbfc580209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Sep 14 13:28:11.146471 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:11.146429    2557 patch_prober.go:28] interesting pod/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Sep 14 13:28:11.146644 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:11.146488    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8" podUID="2e7b8e99-ce68-4bac-9a90-10dbfc580209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Sep 14 13:28:12.146363 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:12.146325    2557 patch_prober.go:28] interesting pod/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Sep 14 13:28:12.146816 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:12.146383    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8" podUID="2e7b8e99-ce68-4bac-9a90-10dbfc580209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Sep 14 13:28:13.151342 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:13.151317    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:28:13.151870 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:13.151720    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:28:13.152370 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:13.152350    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:28:20.440495 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:20.440466    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-622b87-4zrmq"]
Sep 14 13:28:20.444071 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:20.444052    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-622b87-4zrmq"
Sep 14 13:28:20.446781 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:20.446765    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbzp\""
Sep 14 13:28:20.452165 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:20.452144    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-622b87-4zrmq"]
Sep 14 13:28:20.516480 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:20.516446    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bf13c36-2151-4c85-aa5d-d3ec86e2bac3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-622b87-4zrmq\" (UID: \"6bf13c36-2151-4c85-aa5d-d3ec86e2bac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-622b87-4zrmq"
Sep 14 13:28:20.617926 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:20.617893    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bf13c36-2151-4c85-aa5d-d3ec86e2bac3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-622b87-4zrmq\" (UID: \"6bf13c36-2151-4c85-aa5d-d3ec86e2bac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-622b87-4zrmq"
Sep 14 13:28:20.618479 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:20.618461    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bf13c36-2151-4c85-aa5d-d3ec86e2bac3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-622b87-4zrmq\" (UID: \"6bf13c36-2151-4c85-aa5d-d3ec86e2bac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-622b87-4zrmq"
Sep 14 13:28:20.756270 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:20.756196    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-622b87-4zrmq"
Sep 14 13:28:21.035658 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:21.035631    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-622b87-4zrmq"]
Sep 14 13:28:21.037429 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:28:21.037402    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf13c36_2151_4c85_aa5d_d3ec86e2bac3.slice/crio-1b77e788651e7c4bb3c5a8094d55a9e0af76e39ab830d18d943ca7001d6d0e38 WatchSource:0}: Error finding container 1b77e788651e7c4bb3c5a8094d55a9e0af76e39ab830d18d943ca7001d6d0e38: Status 404 returned error can't find the container with id 1b77e788651e7c4bb3c5a8094d55a9e0af76e39ab830d18d943ca7001d6d0e38
Sep 14 13:28:21.511486 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:21.511451    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-622b87-4zrmq" event={"ID":"6bf13c36-2151-4c85-aa5d-d3ec86e2bac3","Type":"ContainerStarted","Data":"838113c53429f884ff7d0dc24b2fd1cd393684442ed4dbebe630b961379b5a7c"}
Sep 14 13:28:21.511881 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:21.511503    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-622b87-4zrmq" event={"ID":"6bf13c36-2151-4c85-aa5d-d3ec86e2bac3","Type":"ContainerStarted","Data":"1b77e788651e7c4bb3c5a8094d55a9e0af76e39ab830d18d943ca7001d6d0e38"}
Sep 14 13:28:21.529805 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:21.529766    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-622b87-4zrmq" podStartSLOduration=1.529728101 podStartE2EDuration="1.529728101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:28:21.529123736 +0000 UTC m=+1819.963799257" watchObservedRunningTime="2026-09-14 13:28:21.529728101 +0000 UTC m=+1819.964403620"
Sep 14 13:28:22.518167 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:22.518132    2557 generic.go:356] "Generic (PLEG): container finished" podID="6bf13c36-2151-4c85-aa5d-d3ec86e2bac3" containerID="838113c53429f884ff7d0dc24b2fd1cd393684442ed4dbebe630b961379b5a7c" exitCode=0
Sep 14 13:28:22.518579 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:22.518196    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-622b87-4zrmq" event={"ID":"6bf13c36-2151-4c85-aa5d-d3ec86e2bac3","Type":"ContainerDied","Data":"838113c53429f884ff7d0dc24b2fd1cd393684442ed4dbebe630b961379b5a7c"}
Sep 14 13:28:23.670388 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:23.670360    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-622b87-4zrmq"
Sep 14 13:28:23.749446 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:23.749419    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bf13c36-2151-4c85-aa5d-d3ec86e2bac3-maas-api-service-ca\") pod \"6bf13c36-2151-4c85-aa5d-d3ec86e2bac3\" (UID: \"6bf13c36-2151-4c85-aa5d-d3ec86e2bac3\") "
Sep 14 13:28:23.749778 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:23.749728    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bf13c36-2151-4c85-aa5d-d3ec86e2bac3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6bf13c36-2151-4c85-aa5d-d3ec86e2bac3" (UID: "6bf13c36-2151-4c85-aa5d-d3ec86e2bac3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:28:23.850971 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:23.850897    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bf13c36-2151-4c85-aa5d-d3ec86e2bac3-maas-api-service-ca\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:28:24.530198 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:24.530163    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-622b87-4zrmq"
Sep 14 13:28:24.530371 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:24.530203    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-622b87-4zrmq" event={"ID":"6bf13c36-2151-4c85-aa5d-d3ec86e2bac3","Type":"ContainerDied","Data":"1b77e788651e7c4bb3c5a8094d55a9e0af76e39ab830d18d943ca7001d6d0e38"}
Sep 14 13:28:24.530371 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:24.530244    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b77e788651e7c4bb3c5a8094d55a9e0af76e39ab830d18d943ca7001d6d0e38"
Sep 14 13:28:35.940038 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:35.940006    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8"]
Sep 14 13:28:35.940480 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:35.940228    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8" podUID="4dcf5d12-54dd-4d49-862c-96c5a874bc74" containerName="payload-processing" containerID="cri-o://5b841d88d7560268fcccf585b9017bc3c65126dd1c9afa03b0c6c19b5a9c4428" gracePeriod=30
Sep 14 13:28:36.128172 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:36.128141    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-622b87-4zrmq"]
Sep 14 13:28:36.130072 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:36.130052    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-622b87-4zrmq"]
Sep 14 13:28:36.213790 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:36.213721    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8"
Sep 14 13:28:36.376711 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:36.376680    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qhtt\" (UniqueName: \"kubernetes.io/projected/4dcf5d12-54dd-4d49-862c-96c5a874bc74-kube-api-access-7qhtt\") pod \"4dcf5d12-54dd-4d49-862c-96c5a874bc74\" (UID: \"4dcf5d12-54dd-4d49-862c-96c5a874bc74\") "
Sep 14 13:28:36.376930 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:36.376776    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dcf5d12-54dd-4d49-862c-96c5a874bc74-plugins-config-volume\") pod \"4dcf5d12-54dd-4d49-862c-96c5a874bc74\" (UID: \"4dcf5d12-54dd-4d49-862c-96c5a874bc74\") "
Sep 14 13:28:36.377131 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:36.377105    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dcf5d12-54dd-4d49-862c-96c5a874bc74-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4dcf5d12-54dd-4d49-862c-96c5a874bc74" (UID: "4dcf5d12-54dd-4d49-862c-96c5a874bc74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:28:36.378897 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:36.378873    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dcf5d12-54dd-4d49-862c-96c5a874bc74-kube-api-access-7qhtt" (OuterVolumeSpecName: "kube-api-access-7qhtt") pod "4dcf5d12-54dd-4d49-862c-96c5a874bc74" (UID: "4dcf5d12-54dd-4d49-862c-96c5a874bc74"). InnerVolumeSpecName "kube-api-access-7qhtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:28:36.477685 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:36.477596    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dcf5d12-54dd-4d49-862c-96c5a874bc74-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:28:36.477685 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:36.477627    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qhtt\" (UniqueName: \"kubernetes.io/projected/4dcf5d12-54dd-4d49-862c-96c5a874bc74-kube-api-access-7qhtt\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:28:36.592275 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:36.592234    2557 generic.go:356] "Generic (PLEG): container finished" podID="4dcf5d12-54dd-4d49-862c-96c5a874bc74" containerID="5b841d88d7560268fcccf585b9017bc3c65126dd1c9afa03b0c6c19b5a9c4428" exitCode=0
Sep 14 13:28:36.592497 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:36.592279    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8" event={"ID":"4dcf5d12-54dd-4d49-862c-96c5a874bc74","Type":"ContainerDied","Data":"5b841d88d7560268fcccf585b9017bc3c65126dd1c9afa03b0c6c19b5a9c4428"}
Sep 14 13:28:36.592497 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:36.592290    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8"
Sep 14 13:28:36.592497 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:36.592339    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8" event={"ID":"4dcf5d12-54dd-4d49-862c-96c5a874bc74","Type":"ContainerDied","Data":"61f387d94c1448715b846d2b0c705f7197ec23d7375bd70ee89a468135834bff"}
Sep 14 13:28:36.592497 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:36.592367    2557 scope.go:117] "RemoveContainer" containerID="5b841d88d7560268fcccf585b9017bc3c65126dd1c9afa03b0c6c19b5a9c4428"
Sep 14 13:28:36.601498 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:36.601477    2557 scope.go:117] "RemoveContainer" containerID="5b841d88d7560268fcccf585b9017bc3c65126dd1c9afa03b0c6c19b5a9c4428"
Sep 14 13:28:36.601799 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:28:36.601782    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b841d88d7560268fcccf585b9017bc3c65126dd1c9afa03b0c6c19b5a9c4428\": container with ID starting with 5b841d88d7560268fcccf585b9017bc3c65126dd1c9afa03b0c6c19b5a9c4428 not found: ID does not exist" containerID="5b841d88d7560268fcccf585b9017bc3c65126dd1c9afa03b0c6c19b5a9c4428"
Sep 14 13:28:36.601863 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:36.601806    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b841d88d7560268fcccf585b9017bc3c65126dd1c9afa03b0c6c19b5a9c4428"} err="failed to get container status \"5b841d88d7560268fcccf585b9017bc3c65126dd1c9afa03b0c6c19b5a9c4428\": rpc error: code = NotFound desc = could not find container \"5b841d88d7560268fcccf585b9017bc3c65126dd1c9afa03b0c6c19b5a9c4428\": container with ID starting with 5b841d88d7560268fcccf585b9017bc3c65126dd1c9afa03b0c6c19b5a9c4428 not found: ID does not exist"
Sep 14 13:28:36.617293 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:36.617266    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8"]
Sep 14 13:28:36.622828 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:36.622804    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-622b87-788c7f5f57-55vd8"]
Sep 14 13:28:38.048917 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:38.048873    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dcf5d12-54dd-4d49-862c-96c5a874bc74" path="/var/lib/kubelet/pods/4dcf5d12-54dd-4d49-862c-96c5a874bc74/volumes"
Sep 14 13:28:38.049413 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:38.049375    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bf13c36-2151-4c85-aa5d-d3ec86e2bac3" path="/var/lib/kubelet/pods/6bf13c36-2151-4c85-aa5d-d3ec86e2bac3/volumes"
Sep 14 13:28:43.800907 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:43.800733    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"]
Sep 14 13:28:43.801420 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:43.801111    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd" podUID="ed06c396-3158-4e8b-a961-89e905914221" containerName="istio-proxy" containerID="cri-o://c790e9ce48f1d369539bd445dc5f9726e53d53f82af82e9264eb1e174b18d4d2" gracePeriod=30
Sep 14 13:28:44.158920 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:44.158836    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cf25e-4sdrx"]
Sep 14 13:28:44.159287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:44.159269    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dcf5d12-54dd-4d49-862c-96c5a874bc74" containerName="payload-processing"
Sep 14 13:28:44.159287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:44.159287    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dcf5d12-54dd-4d49-862c-96c5a874bc74" containerName="payload-processing"
Sep 14 13:28:44.159393 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:44.159297    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bf13c36-2151-4c85-aa5d-d3ec86e2bac3" containerName="revoke-keys"
Sep 14 13:28:44.159393 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:44.159303    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bf13c36-2151-4c85-aa5d-d3ec86e2bac3" containerName="revoke-keys"
Sep 14 13:28:44.159393 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:44.159371    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dcf5d12-54dd-4d49-862c-96c5a874bc74" containerName="payload-processing"
Sep 14 13:28:44.159393 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:44.159383    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bf13c36-2151-4c85-aa5d-d3ec86e2bac3" containerName="revoke-keys"
Sep 14 13:28:44.163985 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:44.163960    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cf25e-4sdrx"
Sep 14 13:28:44.166573 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:44.166549    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbzp\""
Sep 14 13:28:44.172171 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:44.172146    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cf25e-4sdrx"]
Sep 14 13:28:44.250081 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:44.250049    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/598ff86f-261e-4345-92ce-87c123f41a2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1cf25e-4sdrx\" (UID: \"598ff86f-261e-4345-92ce-87c123f41a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cf25e-4sdrx"
Sep 14 13:28:44.351021 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:44.350977    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/598ff86f-261e-4345-92ce-87c123f41a2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1cf25e-4sdrx\" (UID: \"598ff86f-261e-4345-92ce-87c123f41a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cf25e-4sdrx"
Sep 14 13:28:44.351638 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:44.351616    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/598ff86f-261e-4345-92ce-87c123f41a2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1cf25e-4sdrx\" (UID: \"598ff86f-261e-4345-92ce-87c123f41a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cf25e-4sdrx"
Sep 14 13:28:44.476957 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:44.476924    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cf25e-4sdrx"
Sep 14 13:28:44.754699 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:44.754671    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cf25e-4sdrx"]
Sep 14 13:28:44.756835 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:28:44.756813    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598ff86f_261e_4345_92ce_87c123f41a2e.slice/crio-6753b16f6d6f71ffa13405f746e22d84cfe5481cb921dd1a6dfd8a0c039b29c7 WatchSource:0}: Error finding container 6753b16f6d6f71ffa13405f746e22d84cfe5481cb921dd1a6dfd8a0c039b29c7: Status 404 returned error can't find the container with id 6753b16f6d6f71ffa13405f746e22d84cfe5481cb921dd1a6dfd8a0c039b29c7
Sep 14 13:28:45.643294 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:45.643254    2557 generic.go:356] "Generic (PLEG): container finished" podID="598ff86f-261e-4345-92ce-87c123f41a2e" containerID="e16d76db44fab349ef16a4f85cebfd76d61c131941af9812459a24577443152b" exitCode=0
Sep 14 13:28:45.643690 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:45.643314    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cf25e-4sdrx" event={"ID":"598ff86f-261e-4345-92ce-87c123f41a2e","Type":"ContainerDied","Data":"e16d76db44fab349ef16a4f85cebfd76d61c131941af9812459a24577443152b"}
Sep 14 13:28:45.643690 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:45.643364    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cf25e-4sdrx" event={"ID":"598ff86f-261e-4345-92ce-87c123f41a2e","Type":"ContainerStarted","Data":"6753b16f6d6f71ffa13405f746e22d84cfe5481cb921dd1a6dfd8a0c039b29c7"}
Sep 14 13:28:46.790827 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:46.790801    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cf25e-4sdrx"
Sep 14 13:28:46.873880 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:46.873829    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/598ff86f-261e-4345-92ce-87c123f41a2e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "598ff86f-261e-4345-92ce-87c123f41a2e" (UID: "598ff86f-261e-4345-92ce-87c123f41a2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:28:46.874089 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:46.874074    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/598ff86f-261e-4345-92ce-87c123f41a2e-maas-api-service-ca\") pod \"598ff86f-261e-4345-92ce-87c123f41a2e\" (UID: \"598ff86f-261e-4345-92ce-87c123f41a2e\") "
Sep 14 13:28:46.874513 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:46.874494    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/598ff86f-261e-4345-92ce-87c123f41a2e-maas-api-service-ca\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:28:47.655686 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:47.655653    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cf25e-4sdrx" event={"ID":"598ff86f-261e-4345-92ce-87c123f41a2e","Type":"ContainerDied","Data":"6753b16f6d6f71ffa13405f746e22d84cfe5481cb921dd1a6dfd8a0c039b29c7"}
Sep 14 13:28:47.655686 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:47.655675    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cf25e-4sdrx"
Sep 14 13:28:47.655921 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:47.655702    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6753b16f6d6f71ffa13405f746e22d84cfe5481cb921dd1a6dfd8a0c039b29c7"
Sep 14 13:28:49.068179 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.068112    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:28:49.097985 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.097953    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-istio-envoy\") pod \"ed06c396-3158-4e8b-a961-89e905914221\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") "
Sep 14 13:28:49.098140 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.097999    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-workload-socket\") pod \"ed06c396-3158-4e8b-a961-89e905914221\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") "
Sep 14 13:28:49.098140 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.098035    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-istio-data\") pod \"ed06c396-3158-4e8b-a961-89e905914221\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") "
Sep 14 13:28:49.098140 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.098055    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-workload-certs\") pod \"ed06c396-3158-4e8b-a961-89e905914221\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") "
Sep 14 13:28:49.098140 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.098070    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-credential-socket\") pod \"ed06c396-3158-4e8b-a961-89e905914221\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") "
Sep 14 13:28:49.098140 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.098093    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed06c396-3158-4e8b-a961-89e905914221-istio-podinfo\") pod \"ed06c396-3158-4e8b-a961-89e905914221\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") "
Sep 14 13:28:49.098140 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.098137    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4qts\" (UniqueName: \"kubernetes.io/projected/ed06c396-3158-4e8b-a961-89e905914221-kube-api-access-f4qts\") pod \"ed06c396-3158-4e8b-a961-89e905914221\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") "
Sep 14 13:28:49.098484 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.098176    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed06c396-3158-4e8b-a961-89e905914221-istio-token\") pod \"ed06c396-3158-4e8b-a961-89e905914221\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") "
Sep 14 13:28:49.098484 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.098222    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed06c396-3158-4e8b-a961-89e905914221-istiod-ca-cert\") pod \"ed06c396-3158-4e8b-a961-89e905914221\" (UID: \"ed06c396-3158-4e8b-a961-89e905914221\") "
Sep 14 13:28:49.098484 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.098258    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ed06c396-3158-4e8b-a961-89e905914221" (UID: "ed06c396-3158-4e8b-a961-89e905914221"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:28:49.098484 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.098334    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-istio-data" (OuterVolumeSpecName: "istio-data") pod "ed06c396-3158-4e8b-a961-89e905914221" (UID: "ed06c396-3158-4e8b-a961-89e905914221"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:28:49.098484 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.098360    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ed06c396-3158-4e8b-a961-89e905914221" (UID: "ed06c396-3158-4e8b-a961-89e905914221"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:28:49.098782 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.098579    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ed06c396-3158-4e8b-a961-89e905914221" (UID: "ed06c396-3158-4e8b-a961-89e905914221"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:28:49.098782 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.098653    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-workload-socket\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:28:49.098782 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.098669    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed06c396-3158-4e8b-a961-89e905914221-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ed06c396-3158-4e8b-a961-89e905914221" (UID: "ed06c396-3158-4e8b-a961-89e905914221"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:28:49.098782 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.098674    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-istio-data\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:28:49.098782 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.098704    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-workload-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:28:49.098782 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.098718    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-credential-socket\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:28:49.100794 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.100728    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed06c396-3158-4e8b-a961-89e905914221-kube-api-access-f4qts" (OuterVolumeSpecName: "kube-api-access-f4qts") pod "ed06c396-3158-4e8b-a961-89e905914221" (UID: "ed06c396-3158-4e8b-a961-89e905914221"). InnerVolumeSpecName "kube-api-access-f4qts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:28:49.101519 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.101179    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed06c396-3158-4e8b-a961-89e905914221-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ed06c396-3158-4e8b-a961-89e905914221" (UID: "ed06c396-3158-4e8b-a961-89e905914221"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 13:28:49.101519 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.101253    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed06c396-3158-4e8b-a961-89e905914221-istio-token" (OuterVolumeSpecName: "istio-token") pod "ed06c396-3158-4e8b-a961-89e905914221" (UID: "ed06c396-3158-4e8b-a961-89e905914221"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:28:49.101519 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.101449    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ed06c396-3158-4e8b-a961-89e905914221" (UID: "ed06c396-3158-4e8b-a961-89e905914221"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:28:49.199710 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.199673    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed06c396-3158-4e8b-a961-89e905914221-istio-envoy\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:28:49.199710 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.199702    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed06c396-3158-4e8b-a961-89e905914221-istio-podinfo\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:28:49.199710 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.199714    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4qts\" (UniqueName: \"kubernetes.io/projected/ed06c396-3158-4e8b-a961-89e905914221-kube-api-access-f4qts\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:28:49.199987 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.199727    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed06c396-3158-4e8b-a961-89e905914221-istio-token\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:28:49.199987 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.199766    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed06c396-3158-4e8b-a961-89e905914221-istiod-ca-cert\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:28:49.673835 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.673798    2557 generic.go:356] "Generic (PLEG): container finished" podID="ed06c396-3158-4e8b-a961-89e905914221" containerID="c790e9ce48f1d369539bd445dc5f9726e53d53f82af82e9264eb1e174b18d4d2" exitCode=0
Sep 14 13:28:49.674013 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.673855    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"
Sep 14 13:28:49.674013 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.673883    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd" event={"ID":"ed06c396-3158-4e8b-a961-89e905914221","Type":"ContainerDied","Data":"c790e9ce48f1d369539bd445dc5f9726e53d53f82af82e9264eb1e174b18d4d2"}
Sep 14 13:28:49.674013 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.673946    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd" event={"ID":"ed06c396-3158-4e8b-a961-89e905914221","Type":"ContainerDied","Data":"b3e71e27beeda0d305b2821cb8283686dec38b241f86eb6b1a34501653edb5f5"}
Sep 14 13:28:49.674013 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.673970    2557 scope.go:117] "RemoveContainer" containerID="c790e9ce48f1d369539bd445dc5f9726e53d53f82af82e9264eb1e174b18d4d2"
Sep 14 13:28:49.683573 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.683557    2557 scope.go:117] "RemoveContainer" containerID="bf12b2d146f57f01df443b7c28545ebce09acccc8220219ab45ace15840de147"
Sep 14 13:28:49.694824 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.694727    2557 scope.go:117] "RemoveContainer" containerID="c790e9ce48f1d369539bd445dc5f9726e53d53f82af82e9264eb1e174b18d4d2"
Sep 14 13:28:49.695075 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:28:49.695056    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c790e9ce48f1d369539bd445dc5f9726e53d53f82af82e9264eb1e174b18d4d2\": container with ID starting with c790e9ce48f1d369539bd445dc5f9726e53d53f82af82e9264eb1e174b18d4d2 not found: ID does not exist" containerID="c790e9ce48f1d369539bd445dc5f9726e53d53f82af82e9264eb1e174b18d4d2"
Sep 14 13:28:49.695127 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.695084    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c790e9ce48f1d369539bd445dc5f9726e53d53f82af82e9264eb1e174b18d4d2"} err="failed to get container status \"c790e9ce48f1d369539bd445dc5f9726e53d53f82af82e9264eb1e174b18d4d2\": rpc error: code = NotFound desc = could not find container \"c790e9ce48f1d369539bd445dc5f9726e53d53f82af82e9264eb1e174b18d4d2\": container with ID starting with c790e9ce48f1d369539bd445dc5f9726e53d53f82af82e9264eb1e174b18d4d2 not found: ID does not exist"
Sep 14 13:28:49.695127 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.695103    2557 scope.go:117] "RemoveContainer" containerID="bf12b2d146f57f01df443b7c28545ebce09acccc8220219ab45ace15840de147"
Sep 14 13:28:49.695375 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:28:49.695358    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf12b2d146f57f01df443b7c28545ebce09acccc8220219ab45ace15840de147\": container with ID starting with bf12b2d146f57f01df443b7c28545ebce09acccc8220219ab45ace15840de147 not found: ID does not exist" containerID="bf12b2d146f57f01df443b7c28545ebce09acccc8220219ab45ace15840de147"
Sep 14 13:28:49.695420 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.695382    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf12b2d146f57f01df443b7c28545ebce09acccc8220219ab45ace15840de147"} err="failed to get container status \"bf12b2d146f57f01df443b7c28545ebce09acccc8220219ab45ace15840de147\": rpc error: code = NotFound desc = could not find container \"bf12b2d146f57f01df443b7c28545ebce09acccc8220219ab45ace15840de147\": container with ID starting with bf12b2d146f57f01df443b7c28545ebce09acccc8220219ab45ace15840de147 not found: ID does not exist"
Sep 14 13:28:49.696658 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.696638    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"]
Sep 14 13:28:49.701178 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:49.701159    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-622b87-openshift-default-78c7bd45bb-dhptd"]
Sep 14 13:28:50.050825 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:50.050787    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed06c396-3158-4e8b-a961-89e905914221" path="/var/lib/kubelet/pods/ed06c396-3158-4e8b-a961-89e905914221/volumes"
Sep 14 13:28:55.379680 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:55.379643    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7422-e6779f"]
Sep 14 13:28:55.380152 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:55.380076    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed06c396-3158-4e8b-a961-89e905914221" containerName="istio-proxy"
Sep 14 13:28:55.380152 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:55.380088    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed06c396-3158-4e8b-a961-89e905914221" containerName="istio-proxy"
Sep 14 13:28:55.380152 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:55.380099    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed06c396-3158-4e8b-a961-89e905914221" containerName="istio-proxy"
Sep 14 13:28:55.380152 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:55.380105    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed06c396-3158-4e8b-a961-89e905914221" containerName="istio-proxy"
Sep 14 13:28:55.380152 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:55.380139    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="598ff86f-261e-4345-92ce-87c123f41a2e" containerName="revoke-keys"
Sep 14 13:28:55.380152 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:55.380146    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="598ff86f-261e-4345-92ce-87c123f41a2e" containerName="revoke-keys"
Sep 14 13:28:55.380350 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:55.380212    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed06c396-3158-4e8b-a961-89e905914221" containerName="istio-proxy"
Sep 14 13:28:55.380350 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:55.380221    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="598ff86f-261e-4345-92ce-87c123f41a2e" containerName="revoke-keys"
Sep 14 13:28:55.380350 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:55.380226    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed06c396-3158-4e8b-a961-89e905914221" containerName="istio-proxy"
Sep 14 13:28:55.384899 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:55.384880    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-e6779f"
Sep 14 13:28:55.387793 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:55.387767    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vtxmg\""
Sep 14 13:28:55.389678 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:55.389657    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7422-e6779f"]
Sep 14 13:28:55.456635 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:55.456599    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz4dd\" (UniqueName: \"kubernetes.io/projected/24fab010-7b0b-4b5e-9432-9733b2425f83-kube-api-access-kz4dd\") pod \"test-curl-7422-e6779f\" (UID: \"24fab010-7b0b-4b5e-9432-9733b2425f83\") " pod="opendatahub/test-curl-7422-e6779f"
Sep 14 13:28:55.558118 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:55.558079    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz4dd\" (UniqueName: \"kubernetes.io/projected/24fab010-7b0b-4b5e-9432-9733b2425f83-kube-api-access-kz4dd\") pod \"test-curl-7422-e6779f\" (UID: \"24fab010-7b0b-4b5e-9432-9733b2425f83\") " pod="opendatahub/test-curl-7422-e6779f"
Sep 14 13:28:55.568493 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:55.568461    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz4dd\" (UniqueName: \"kubernetes.io/projected/24fab010-7b0b-4b5e-9432-9733b2425f83-kube-api-access-kz4dd\") pod \"test-curl-7422-e6779f\" (UID: \"24fab010-7b0b-4b5e-9432-9733b2425f83\") " pod="opendatahub/test-curl-7422-e6779f"
Sep 14 13:28:55.695656 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:55.695628    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-e6779f"
Sep 14 13:28:55.966890 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:55.966864    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7422-e6779f"]
Sep 14 13:28:55.968732 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:28:55.968699    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24fab010_7b0b_4b5e_9432_9733b2425f83.slice/crio-27b17aee6c2402215ba248db9a9ab2d8d0d940153497352be99b8e426807edf2 WatchSource:0}: Error finding container 27b17aee6c2402215ba248db9a9ab2d8d0d940153497352be99b8e426807edf2: Status 404 returned error can't find the container with id 27b17aee6c2402215ba248db9a9ab2d8d0d940153497352be99b8e426807edf2
Sep 14 13:28:55.970733 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:55.970714    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 13:28:56.713984 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:56.713944    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7422-e6779f" event={"ID":"24fab010-7b0b-4b5e-9432-9733b2425f83","Type":"ContainerStarted","Data":"27b17aee6c2402215ba248db9a9ab2d8d0d940153497352be99b8e426807edf2"}
Sep 14 13:28:58.727855 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:58.727816    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7422-e6779f" event={"ID":"24fab010-7b0b-4b5e-9432-9733b2425f83","Type":"ContainerStarted","Data":"d200677eeb6aedd01b81b3801a242e0f9b1179db248c2f954fdd7cff75320e13"}
Sep 14 13:28:58.745167 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:58.745117    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7422-e6779f" podStartSLOduration=1.736456813 podStartE2EDuration="3.745101268s" totalImagesPullingTime="2.008644455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:28:58.743574971 +0000 UTC m=+1857.178250492" watchObservedRunningTime="2026-09-14 13:28:58.745101268 +0000 UTC m=+1857.179776786"
Sep 14 13:28:59.368302 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:59.368268    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7422-e6779f"]
Sep 14 13:28:59.372768 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:59.372723    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7422-e6779f"]
Sep 14 13:28:59.491085 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:59.491050    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7422-f7aeb0"]
Sep 14 13:28:59.491546 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:59.491526    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24fab010-7b0b-4b5e-9432-9733b2425f83" containerName="test-curl-7422-e6779f"
Sep 14 13:28:59.491546 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:59.491546    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="24fab010-7b0b-4b5e-9432-9733b2425f83" containerName="test-curl-7422-e6779f"
Sep 14 13:28:59.491724 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:59.491625    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="24fab010-7b0b-4b5e-9432-9733b2425f83" containerName="test-curl-7422-e6779f"
Sep 14 13:28:59.494803 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:59.494788    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-f7aeb0"
Sep 14 13:28:59.499778 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:59.499756    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7422-f7aeb0"]
Sep 14 13:28:59.591893 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:59.591847    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbsw2\" (UniqueName: \"kubernetes.io/projected/a4eef52d-4183-4167-bb22-dcc27e891d46-kube-api-access-kbsw2\") pod \"test-curl-7422-f7aeb0\" (UID: \"a4eef52d-4183-4167-bb22-dcc27e891d46\") " pod="opendatahub/test-curl-7422-f7aeb0"
Sep 14 13:28:59.692362 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:59.692325    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbsw2\" (UniqueName: \"kubernetes.io/projected/a4eef52d-4183-4167-bb22-dcc27e891d46-kube-api-access-kbsw2\") pod \"test-curl-7422-f7aeb0\" (UID: \"a4eef52d-4183-4167-bb22-dcc27e891d46\") " pod="opendatahub/test-curl-7422-f7aeb0"
Sep 14 13:28:59.702201 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:59.702176    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbsw2\" (UniqueName: \"kubernetes.io/projected/a4eef52d-4183-4167-bb22-dcc27e891d46-kube-api-access-kbsw2\") pod \"test-curl-7422-f7aeb0\" (UID: \"a4eef52d-4183-4167-bb22-dcc27e891d46\") " pod="opendatahub/test-curl-7422-f7aeb0"
Sep 14 13:28:59.733408 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:28:59.733378    2557 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7422-e6779f" container="test-curl-7422-e6779f"
Sep 14 13:28:59.805596 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:28:59.805522    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-f7aeb0"
Sep 14 13:29:00.053211 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.053177    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j"]
Sep 14 13:29:00.053508 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.053433    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j" podUID="e4fd4938-5f6d-49b7-9909-a3bad54689b4" containerName="payload-pre-processing" containerID="cri-o://2b8c4ca64b46d1d755adf3f108d466b3ac373179b4c4aab4ac93a5f03f91e79a" gracePeriod=30
Sep 14 13:29:00.080357 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.080329    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7422-f7aeb0"]
Sep 14 13:29:00.119060 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:29:00.119023    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4eef52d_4183_4167_bb22_dcc27e891d46.slice/crio-6b8e3a3bd34b140d4006b6f3d4e0908caf99c0923a73e0bd04089d9add6ed1d4 WatchSource:0}: Error finding container 6b8e3a3bd34b140d4006b6f3d4e0908caf99c0923a73e0bd04089d9add6ed1d4: Status 404 returned error can't find the container with id 6b8e3a3bd34b140d4006b6f3d4e0908caf99c0923a73e0bd04089d9add6ed1d4
Sep 14 13:29:00.190566 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.190540    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cf25e-4sdrx"]
Sep 14 13:29:00.193917 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.193895    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cf25e-4sdrx"]
Sep 14 13:29:00.316362 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.316293    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j"
Sep 14 13:29:00.400420 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.400381    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4fd4938-5f6d-49b7-9909-a3bad54689b4-plugins-config-volume\") pod \"e4fd4938-5f6d-49b7-9909-a3bad54689b4\" (UID: \"e4fd4938-5f6d-49b7-9909-a3bad54689b4\") "
Sep 14 13:29:00.400420 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.400426    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvzld\" (UniqueName: \"kubernetes.io/projected/e4fd4938-5f6d-49b7-9909-a3bad54689b4-kube-api-access-fvzld\") pod \"e4fd4938-5f6d-49b7-9909-a3bad54689b4\" (UID: \"e4fd4938-5f6d-49b7-9909-a3bad54689b4\") "
Sep 14 13:29:00.400783 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.400760    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4fd4938-5f6d-49b7-9909-a3bad54689b4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4fd4938-5f6d-49b7-9909-a3bad54689b4" (UID: "e4fd4938-5f6d-49b7-9909-a3bad54689b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:29:00.402368 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.402343    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4fd4938-5f6d-49b7-9909-a3bad54689b4-kube-api-access-fvzld" (OuterVolumeSpecName: "kube-api-access-fvzld") pod "e4fd4938-5f6d-49b7-9909-a3bad54689b4" (UID: "e4fd4938-5f6d-49b7-9909-a3bad54689b4"). InnerVolumeSpecName "kube-api-access-fvzld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:29:00.501855 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.501823    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4fd4938-5f6d-49b7-9909-a3bad54689b4-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:29:00.501855 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.501850    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvzld\" (UniqueName: \"kubernetes.io/projected/e4fd4938-5f6d-49b7-9909-a3bad54689b4-kube-api-access-fvzld\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:29:00.740813 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.740774    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7422-f7aeb0" event={"ID":"a4eef52d-4183-4167-bb22-dcc27e891d46","Type":"ContainerStarted","Data":"4c3df81e06bc01dcfb15c1e9bfc714a6648fd95fff1fefe7797b9d2c51917722"}
Sep 14 13:29:00.741268 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.740828    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7422-f7aeb0" event={"ID":"a4eef52d-4183-4167-bb22-dcc27e891d46","Type":"ContainerStarted","Data":"6b8e3a3bd34b140d4006b6f3d4e0908caf99c0923a73e0bd04089d9add6ed1d4"}
Sep 14 13:29:00.742232 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.742204    2557 generic.go:356] "Generic (PLEG): container finished" podID="e4fd4938-5f6d-49b7-9909-a3bad54689b4" containerID="2b8c4ca64b46d1d755adf3f108d466b3ac373179b4c4aab4ac93a5f03f91e79a" exitCode=0
Sep 14 13:29:00.742336 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.742236    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j"
Sep 14 13:29:00.742336 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.742273    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j" event={"ID":"e4fd4938-5f6d-49b7-9909-a3bad54689b4","Type":"ContainerDied","Data":"2b8c4ca64b46d1d755adf3f108d466b3ac373179b4c4aab4ac93a5f03f91e79a"}
Sep 14 13:29:00.742336 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.742316    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j" event={"ID":"e4fd4938-5f6d-49b7-9909-a3bad54689b4","Type":"ContainerDied","Data":"fce0f2337c572180b787622edbc780c34bbefb2a97703f074b60ff7a4c4b18b1"}
Sep 14 13:29:00.742438 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.742340    2557 scope.go:117] "RemoveContainer" containerID="2b8c4ca64b46d1d755adf3f108d466b3ac373179b4c4aab4ac93a5f03f91e79a"
Sep 14 13:29:00.742479 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.742465    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7422-e6779f" podUID="24fab010-7b0b-4b5e-9432-9733b2425f83" containerName="test-curl-7422-e6779f" containerID="cri-o://d200677eeb6aedd01b81b3801a242e0f9b1179db248c2f954fdd7cff75320e13" gracePeriod=2
Sep 14 13:29:00.754937 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.753898    2557 scope.go:117] "RemoveContainer" containerID="2b8c4ca64b46d1d755adf3f108d466b3ac373179b4c4aab4ac93a5f03f91e79a"
Sep 14 13:29:00.755516 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:29:00.755474    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b8c4ca64b46d1d755adf3f108d466b3ac373179b4c4aab4ac93a5f03f91e79a\": container with ID starting with 2b8c4ca64b46d1d755adf3f108d466b3ac373179b4c4aab4ac93a5f03f91e79a not found: ID does not exist" containerID="2b8c4ca64b46d1d755adf3f108d466b3ac373179b4c4aab4ac93a5f03f91e79a"
Sep 14 13:29:00.755627 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.755515    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b8c4ca64b46d1d755adf3f108d466b3ac373179b4c4aab4ac93a5f03f91e79a"} err="failed to get container status \"2b8c4ca64b46d1d755adf3f108d466b3ac373179b4c4aab4ac93a5f03f91e79a\": rpc error: code = NotFound desc = could not find container \"2b8c4ca64b46d1d755adf3f108d466b3ac373179b4c4aab4ac93a5f03f91e79a\": container with ID starting with 2b8c4ca64b46d1d755adf3f108d466b3ac373179b4c4aab4ac93a5f03f91e79a not found: ID does not exist"
Sep 14 13:29:00.756918 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.756867    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7422-f7aeb0" podStartSLOduration=1.7568486 podStartE2EDuration="1.7568486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:29:00.755853736 +0000 UTC m=+1859.190529256" watchObservedRunningTime="2026-09-14 13:29:00.7568486 +0000 UTC m=+1859.191524119"
Sep 14 13:29:00.769208 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.769187    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j"]
Sep 14 13:29:00.773917 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:00.773899    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1cf25e-747bc8695d-lhj8j"]
Sep 14 13:29:01.396863 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:01.396832    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7422-f7aeb0"]
Sep 14 13:29:01.405132 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:01.405099    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7422-f7aeb0"]
Sep 14 13:29:01.748701 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:29:01.748675    2557 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7422-f7aeb0" container="test-curl-7422-f7aeb0"
Sep 14 13:29:02.049022 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:02.048948    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="598ff86f-261e-4345-92ce-87c123f41a2e" path="/var/lib/kubelet/pods/598ff86f-261e-4345-92ce-87c123f41a2e/volumes"
Sep 14 13:29:02.049322 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:02.049308    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4fd4938-5f6d-49b7-9909-a3bad54689b4" path="/var/lib/kubelet/pods/e4fd4938-5f6d-49b7-9909-a3bad54689b4/volumes"
Sep 14 13:29:02.526473 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:02.526437    2557 scope.go:117] "RemoveContainer" containerID="beb0921f5e17bf627f66d6ed0f79d3dc3b211d0713b04ac77783137c68078792"
Sep 14 13:29:02.753804 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:02.753771    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7422-f7aeb0" podUID="a4eef52d-4183-4167-bb22-dcc27e891d46" containerName="test-curl-7422-f7aeb0" containerID="cri-o://4c3df81e06bc01dcfb15c1e9bfc714a6648fd95fff1fefe7797b9d2c51917722" gracePeriod=2
Sep 14 13:29:02.939729 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:02.939702    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-e6779f"
Sep 14 13:29:03.027173 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:03.027137    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kz4dd\" (UniqueName: \"kubernetes.io/projected/24fab010-7b0b-4b5e-9432-9733b2425f83-kube-api-access-kz4dd\") pod \"24fab010-7b0b-4b5e-9432-9733b2425f83\" (UID: \"24fab010-7b0b-4b5e-9432-9733b2425f83\") "
Sep 14 13:29:03.029312 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:03.029284    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24fab010-7b0b-4b5e-9432-9733b2425f83-kube-api-access-kz4dd" (OuterVolumeSpecName: "kube-api-access-kz4dd") pod "24fab010-7b0b-4b5e-9432-9733b2425f83" (UID: "24fab010-7b0b-4b5e-9432-9733b2425f83"). InnerVolumeSpecName "kube-api-access-kz4dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:29:03.128817 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:03.128785    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz4dd\" (UniqueName: \"kubernetes.io/projected/24fab010-7b0b-4b5e-9432-9733b2425f83-kube-api-access-kz4dd\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:29:03.761877 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:03.761840    2557 generic.go:356] "Generic (PLEG): container finished" podID="24fab010-7b0b-4b5e-9432-9733b2425f83" containerID="d200677eeb6aedd01b81b3801a242e0f9b1179db248c2f954fdd7cff75320e13" exitCode=137
Sep 14 13:29:03.761877 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:03.761881    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-e6779f"
Sep 14 13:29:03.762447 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:03.761950    2557 scope.go:117] "RemoveContainer" containerID="d200677eeb6aedd01b81b3801a242e0f9b1179db248c2f954fdd7cff75320e13"
Sep 14 13:29:03.771416 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:03.771400    2557 scope.go:117] "RemoveContainer" containerID="d200677eeb6aedd01b81b3801a242e0f9b1179db248c2f954fdd7cff75320e13"
Sep 14 13:29:03.771719 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:29:03.771696    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d200677eeb6aedd01b81b3801a242e0f9b1179db248c2f954fdd7cff75320e13\": container with ID starting with d200677eeb6aedd01b81b3801a242e0f9b1179db248c2f954fdd7cff75320e13 not found: ID does not exist" containerID="d200677eeb6aedd01b81b3801a242e0f9b1179db248c2f954fdd7cff75320e13"
Sep 14 13:29:03.771830 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:03.771729    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d200677eeb6aedd01b81b3801a242e0f9b1179db248c2f954fdd7cff75320e13"} err="failed to get container status \"d200677eeb6aedd01b81b3801a242e0f9b1179db248c2f954fdd7cff75320e13\": rpc error: code = NotFound desc = could not find container \"d200677eeb6aedd01b81b3801a242e0f9b1179db248c2f954fdd7cff75320e13\": container with ID starting with d200677eeb6aedd01b81b3801a242e0f9b1179db248c2f954fdd7cff75320e13 not found: ID does not exist"
Sep 14 13:29:03.772889 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:29:03.772860    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7422-e6779f\" is forbidden: User \"system:node:ip-10-0-129-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-61.ec2.internal' and this object" podUID="24fab010-7b0b-4b5e-9432-9733b2425f83" pod="opendatahub/test-curl-7422-e6779f"
Sep 14 13:29:04.049078 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:04.048998    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24fab010-7b0b-4b5e-9432-9733b2425f83" path="/var/lib/kubelet/pods/24fab010-7b0b-4b5e-9432-9733b2425f83/volumes"
Sep 14 13:29:04.952402 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:04.952376    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-f7aeb0"
Sep 14 13:29:05.047305 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:05.047221    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbsw2\" (UniqueName: \"kubernetes.io/projected/a4eef52d-4183-4167-bb22-dcc27e891d46-kube-api-access-kbsw2\") pod \"a4eef52d-4183-4167-bb22-dcc27e891d46\" (UID: \"a4eef52d-4183-4167-bb22-dcc27e891d46\") "
Sep 14 13:29:05.049338 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:05.049303    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4eef52d-4183-4167-bb22-dcc27e891d46-kube-api-access-kbsw2" (OuterVolumeSpecName: "kube-api-access-kbsw2") pod "a4eef52d-4183-4167-bb22-dcc27e891d46" (UID: "a4eef52d-4183-4167-bb22-dcc27e891d46"). InnerVolumeSpecName "kube-api-access-kbsw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:29:05.148485 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:05.148454    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbsw2\" (UniqueName: \"kubernetes.io/projected/a4eef52d-4183-4167-bb22-dcc27e891d46-kube-api-access-kbsw2\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:29:05.774345 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:05.774311    2557 generic.go:356] "Generic (PLEG): container finished" podID="a4eef52d-4183-4167-bb22-dcc27e891d46" containerID="4c3df81e06bc01dcfb15c1e9bfc714a6648fd95fff1fefe7797b9d2c51917722" exitCode=137
Sep 14 13:29:05.774503 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:05.774354    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-f7aeb0"
Sep 14 13:29:05.774503 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:05.774417    2557 scope.go:117] "RemoveContainer" containerID="4c3df81e06bc01dcfb15c1e9bfc714a6648fd95fff1fefe7797b9d2c51917722"
Sep 14 13:29:05.783405 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:05.783389    2557 scope.go:117] "RemoveContainer" containerID="4c3df81e06bc01dcfb15c1e9bfc714a6648fd95fff1fefe7797b9d2c51917722"
Sep 14 13:29:05.783661 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:29:05.783644    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c3df81e06bc01dcfb15c1e9bfc714a6648fd95fff1fefe7797b9d2c51917722\": container with ID starting with 4c3df81e06bc01dcfb15c1e9bfc714a6648fd95fff1fefe7797b9d2c51917722 not found: ID does not exist" containerID="4c3df81e06bc01dcfb15c1e9bfc714a6648fd95fff1fefe7797b9d2c51917722"
Sep 14 13:29:05.783728 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:05.783675    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c3df81e06bc01dcfb15c1e9bfc714a6648fd95fff1fefe7797b9d2c51917722"} err="failed to get container status \"4c3df81e06bc01dcfb15c1e9bfc714a6648fd95fff1fefe7797b9d2c51917722\": rpc error: code = NotFound desc = could not find container \"4c3df81e06bc01dcfb15c1e9bfc714a6648fd95fff1fefe7797b9d2c51917722\": container with ID starting with 4c3df81e06bc01dcfb15c1e9bfc714a6648fd95fff1fefe7797b9d2c51917722 not found: ID does not exist"
Sep 14 13:29:05.785201 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:29:05.785177    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7422-f7aeb0\" is forbidden: User \"system:node:ip-10-0-129-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-61.ec2.internal' and this object" podUID="a4eef52d-4183-4167-bb22-dcc27e891d46" pod="opendatahub/test-curl-7422-f7aeb0"
Sep 14 13:29:06.048641 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:06.048551    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4eef52d-4183-4167-bb22-dcc27e891d46" path="/var/lib/kubelet/pods/a4eef52d-4183-4167-bb22-dcc27e891d46/volumes"
Sep 14 13:29:07.675445 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:07.675417    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"]
Sep 14 13:29:07.675865 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:07.675614    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8" podUID="2e7b8e99-ce68-4bac-9a90-10dbfc580209" containerName="istio-proxy" containerID="cri-o://da66ecc96d8ea09c3fc8d05cbfab47f66b563ae5bb13499a8f76909f4b615d7a" gracePeriod=30
Sep 14 13:29:07.997184 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:07.997152    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-31063a-hssvf"]
Sep 14 13:29:07.997660 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:07.997639    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4fd4938-5f6d-49b7-9909-a3bad54689b4" containerName="payload-pre-processing"
Sep 14 13:29:07.997660 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:07.997658    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4fd4938-5f6d-49b7-9909-a3bad54689b4" containerName="payload-pre-processing"
Sep 14 13:29:07.997903 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:07.997674    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4eef52d-4183-4167-bb22-dcc27e891d46" containerName="test-curl-7422-f7aeb0"
Sep 14 13:29:07.997903 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:07.997679    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4eef52d-4183-4167-bb22-dcc27e891d46" containerName="test-curl-7422-f7aeb0"
Sep 14 13:29:07.997903 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:07.997773    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4fd4938-5f6d-49b7-9909-a3bad54689b4" containerName="payload-pre-processing"
Sep 14 13:29:07.997903 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:07.997783    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4eef52d-4183-4167-bb22-dcc27e891d46" containerName="test-curl-7422-f7aeb0"
Sep 14 13:29:08.002149 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:08.002130    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-31063a-hssvf"
Sep 14 13:29:08.004907 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:08.004886    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbzp\""
Sep 14 13:29:08.008994 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:08.008971    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-31063a-hssvf"]
Sep 14 13:29:08.075161 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:08.075133    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c93d64-afc7-4a87-a13e-14eee62ed936-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-31063a-hssvf\" (UID: \"53c93d64-afc7-4a87-a13e-14eee62ed936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-31063a-hssvf"
Sep 14 13:29:08.176087 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:08.176050    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c93d64-afc7-4a87-a13e-14eee62ed936-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-31063a-hssvf\" (UID: \"53c93d64-afc7-4a87-a13e-14eee62ed936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-31063a-hssvf"
Sep 14 13:29:08.176640 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:08.176620    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c93d64-afc7-4a87-a13e-14eee62ed936-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-31063a-hssvf\" (UID: \"53c93d64-afc7-4a87-a13e-14eee62ed936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-31063a-hssvf"
Sep 14 13:29:08.315197 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:08.315099    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-31063a-hssvf"
Sep 14 13:29:08.589969 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:08.589946    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-31063a-hssvf"]
Sep 14 13:29:08.592197 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:29:08.592170    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c93d64_afc7_4a87_a13e_14eee62ed936.slice/crio-96ebbc57b25c15e0d78c474ed7fcbae00842ce0a171c8075db5911e390a840f2 WatchSource:0}: Error finding container 96ebbc57b25c15e0d78c474ed7fcbae00842ce0a171c8075db5911e390a840f2: Status 404 returned error can't find the container with id 96ebbc57b25c15e0d78c474ed7fcbae00842ce0a171c8075db5911e390a840f2
Sep 14 13:29:08.793789 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:08.793727    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-31063a-hssvf" event={"ID":"53c93d64-afc7-4a87-a13e-14eee62ed936","Type":"ContainerStarted","Data":"8d2eaca4f63b8033ad3f8d3ccc0fb5783f9750c0a346a8afd20872a0ee1827af"}
Sep 14 13:29:08.794166 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:08.793814    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-31063a-hssvf" event={"ID":"53c93d64-afc7-4a87-a13e-14eee62ed936","Type":"ContainerStarted","Data":"96ebbc57b25c15e0d78c474ed7fcbae00842ce0a171c8075db5911e390a840f2"}
Sep 14 13:29:08.812680 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:08.812628    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-31063a-hssvf" podStartSLOduration=1.8126136320000001 podStartE2EDuration="1.812613632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:29:08.810032701 +0000 UTC m=+1867.244708231" watchObservedRunningTime="2026-09-14 13:29:08.812613632 +0000 UTC m=+1867.247289152"
Sep 14 13:29:09.800794 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:09.800733    2557 generic.go:356] "Generic (PLEG): container finished" podID="53c93d64-afc7-4a87-a13e-14eee62ed936" containerID="8d2eaca4f63b8033ad3f8d3ccc0fb5783f9750c0a346a8afd20872a0ee1827af" exitCode=0
Sep 14 13:29:09.801303 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:09.800839    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-31063a-hssvf" event={"ID":"53c93d64-afc7-4a87-a13e-14eee62ed936","Type":"ContainerDied","Data":"8d2eaca4f63b8033ad3f8d3ccc0fb5783f9750c0a346a8afd20872a0ee1827af"}
Sep 14 13:29:10.948394 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:10.948371    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-31063a-hssvf"
Sep 14 13:29:11.101469 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:11.101394    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c93d64-afc7-4a87-a13e-14eee62ed936-maas-api-service-ca\") pod \"53c93d64-afc7-4a87-a13e-14eee62ed936\" (UID: \"53c93d64-afc7-4a87-a13e-14eee62ed936\") "
Sep 14 13:29:11.101768 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:11.101728    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c93d64-afc7-4a87-a13e-14eee62ed936-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "53c93d64-afc7-4a87-a13e-14eee62ed936" (UID: "53c93d64-afc7-4a87-a13e-14eee62ed936"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:29:11.202954 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:11.202923    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c93d64-afc7-4a87-a13e-14eee62ed936-maas-api-service-ca\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:29:11.813036 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:11.813003    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-31063a-hssvf"
Sep 14 13:29:11.813208 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:11.813043    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-31063a-hssvf" event={"ID":"53c93d64-afc7-4a87-a13e-14eee62ed936","Type":"ContainerDied","Data":"96ebbc57b25c15e0d78c474ed7fcbae00842ce0a171c8075db5911e390a840f2"}
Sep 14 13:29:11.813208 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:11.813088    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96ebbc57b25c15e0d78c474ed7fcbae00842ce0a171c8075db5911e390a840f2"
Sep 14 13:29:12.822162 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:12.822132    2557 generic.go:356] "Generic (PLEG): container finished" podID="2e7b8e99-ce68-4bac-9a90-10dbfc580209" containerID="da66ecc96d8ea09c3fc8d05cbfab47f66b563ae5bb13499a8f76909f4b615d7a" exitCode=0
Sep 14 13:29:12.822538 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:12.822218    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8" event={"ID":"2e7b8e99-ce68-4bac-9a90-10dbfc580209","Type":"ContainerDied","Data":"da66ecc96d8ea09c3fc8d05cbfab47f66b563ae5bb13499a8f76909f4b615d7a"}
Sep 14 13:29:12.822538 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:12.822289    2557 scope.go:117] "RemoveContainer" containerID="7aff163bab6c22e5ebcff9a154cb9fe9a7c5fde683df2ca27a6a2a14a3db15b2"
Sep 14 13:29:12.932603 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:12.932578    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:29:13.020631 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.020596    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-token\") pod \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") "
Sep 14 13:29:13.020858 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.020677    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzjcd\" (UniqueName: \"kubernetes.io/projected/2e7b8e99-ce68-4bac-9a90-10dbfc580209-kube-api-access-fzjcd\") pod \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") "
Sep 14 13:29:13.020858 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.020715    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-podinfo\") pod \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") "
Sep 14 13:29:13.020858 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.020775    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-data\") pod \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") "
Sep 14 13:29:13.021099 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.021078    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-credential-socket\") pod \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") "
Sep 14 13:29:13.021236 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.021206    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-workload-socket\") pod \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") "
Sep 14 13:29:13.021363 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.021113    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-data" (OuterVolumeSpecName: "istio-data") pod "2e7b8e99-ce68-4bac-9a90-10dbfc580209" (UID: "2e7b8e99-ce68-4bac-9a90-10dbfc580209"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:29:13.021363 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.021294    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istiod-ca-cert\") pod \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") "
Sep 14 13:29:13.021363 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.021319    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-workload-certs\") pod \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") "
Sep 14 13:29:13.021542 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.021230    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2e7b8e99-ce68-4bac-9a90-10dbfc580209" (UID: "2e7b8e99-ce68-4bac-9a90-10dbfc580209"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:29:13.021542 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.021412    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-envoy\") pod \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\" (UID: \"2e7b8e99-ce68-4bac-9a90-10dbfc580209\") "
Sep 14 13:29:13.021649 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.021615    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2e7b8e99-ce68-4bac-9a90-10dbfc580209" (UID: "2e7b8e99-ce68-4bac-9a90-10dbfc580209"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:29:13.021911 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.021875    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2e7b8e99-ce68-4bac-9a90-10dbfc580209" (UID: "2e7b8e99-ce68-4bac-9a90-10dbfc580209"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:29:13.021988 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.021912    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-data\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:29:13.021988 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.021931    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-credential-socket\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:29:13.021988 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.021943    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istiod-ca-cert\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:29:13.021988 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.021939    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2e7b8e99-ce68-4bac-9a90-10dbfc580209" (UID: "2e7b8e99-ce68-4bac-9a90-10dbfc580209"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:29:13.023070 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.023051    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-token" (OuterVolumeSpecName: "istio-token") pod "2e7b8e99-ce68-4bac-9a90-10dbfc580209" (UID: "2e7b8e99-ce68-4bac-9a90-10dbfc580209"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:29:13.023136 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.023087    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2e7b8e99-ce68-4bac-9a90-10dbfc580209" (UID: "2e7b8e99-ce68-4bac-9a90-10dbfc580209"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 13:29:13.023373 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.023356    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7b8e99-ce68-4bac-9a90-10dbfc580209-kube-api-access-fzjcd" (OuterVolumeSpecName: "kube-api-access-fzjcd") pod "2e7b8e99-ce68-4bac-9a90-10dbfc580209" (UID: "2e7b8e99-ce68-4bac-9a90-10dbfc580209"). InnerVolumeSpecName "kube-api-access-fzjcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:29:13.023501 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.023478    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2e7b8e99-ce68-4bac-9a90-10dbfc580209" (UID: "2e7b8e99-ce68-4bac-9a90-10dbfc580209"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:29:13.123030 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.122997    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzjcd\" (UniqueName: \"kubernetes.io/projected/2e7b8e99-ce68-4bac-9a90-10dbfc580209-kube-api-access-fzjcd\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:29:13.123030 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.123025    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-podinfo\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:29:13.123030 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.123037    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-workload-socket\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:29:13.123251 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.123048    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-workload-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:29:13.123251 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.123059    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-envoy\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:29:13.123251 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.123068    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e7b8e99-ce68-4bac-9a90-10dbfc580209-istio-token\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:29:13.829080 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.829050    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8" event={"ID":"2e7b8e99-ce68-4bac-9a90-10dbfc580209","Type":"ContainerDied","Data":"193345ab3405120fd10d1ca57c754764155ee09bcb1ff87d004d7ca1bac89563"}
Sep 14 13:29:13.829080 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.829075    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"
Sep 14 13:29:13.829558 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.829109    2557 scope.go:117] "RemoveContainer" containerID="da66ecc96d8ea09c3fc8d05cbfab47f66b563ae5bb13499a8f76909f4b615d7a"
Sep 14 13:29:13.852399 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.852370    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"]
Sep 14 13:29:13.856580 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:13.856558    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1cf25e-openshift-default-7479f9bfbb-cjbz8"]
Sep 14 13:29:14.049105 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:14.049072    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e7b8e99-ce68-4bac-9a90-10dbfc580209" path="/var/lib/kubelet/pods/2e7b8e99-ce68-4bac-9a90-10dbfc580209/volumes"
Sep 14 13:29:18.572870 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:18.572835    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9"]
Sep 14 13:29:18.573367 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:18.573275    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9" podUID="0d1a61ee-a98b-4d49-87e9-6700bce75a1c" containerName="payload-processing" containerID="cri-o://afea69578344fff65dfb420f615e89c17ef79f7d6b13330df2a476bce8af1870" gracePeriod=30
Sep 14 13:29:18.581929 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:18.581902    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2"]
Sep 14 13:29:18.582171 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:18.582141    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2" podUID="d272c237-fddc-42d5-ad6f-c222c3370b96" containerName="payload-pre-processing" containerID="cri-o://2c4d3c53bf32efe1c18bd7a7facfb588a47aeb7fc512bc999cbbcbe447e73b51" gracePeriod=30
Sep 14 13:29:18.741917 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:18.741883    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-31063a-hssvf"]
Sep 14 13:29:18.747898 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:18.747285    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-31063a-hssvf"]
Sep 14 13:29:18.859568 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:18.859541    2557 generic.go:356] "Generic (PLEG): container finished" podID="0d1a61ee-a98b-4d49-87e9-6700bce75a1c" containerID="afea69578344fff65dfb420f615e89c17ef79f7d6b13330df2a476bce8af1870" exitCode=0
Sep 14 13:29:18.859703 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:18.859618    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9" event={"ID":"0d1a61ee-a98b-4d49-87e9-6700bce75a1c","Type":"ContainerDied","Data":"afea69578344fff65dfb420f615e89c17ef79f7d6b13330df2a476bce8af1870"}
Sep 14 13:29:18.859703 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:18.859686    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9" event={"ID":"0d1a61ee-a98b-4d49-87e9-6700bce75a1c","Type":"ContainerDied","Data":"a002f3ac51e6dc54e3102e2ef7447ad60d58451f1670fd87b8645a34335d4331"}
Sep 14 13:29:18.859845 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:18.859717    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a002f3ac51e6dc54e3102e2ef7447ad60d58451f1670fd87b8645a34335d4331"
Sep 14 13:29:18.861223 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:18.861196    2557 generic.go:356] "Generic (PLEG): container finished" podID="d272c237-fddc-42d5-ad6f-c222c3370b96" containerID="2c4d3c53bf32efe1c18bd7a7facfb588a47aeb7fc512bc999cbbcbe447e73b51" exitCode=0
Sep 14 13:29:18.861363 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:18.861266    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2" event={"ID":"d272c237-fddc-42d5-ad6f-c222c3370b96","Type":"ContainerDied","Data":"2c4d3c53bf32efe1c18bd7a7facfb588a47aeb7fc512bc999cbbcbe447e73b51"}
Sep 14 13:29:18.882542 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:18.882516    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9"
Sep 14 13:29:18.885415 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:18.885394    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2"
Sep 14 13:29:19.076952 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:19.076910    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5h8x\" (UniqueName: \"kubernetes.io/projected/0d1a61ee-a98b-4d49-87e9-6700bce75a1c-kube-api-access-d5h8x\") pod \"0d1a61ee-a98b-4d49-87e9-6700bce75a1c\" (UID: \"0d1a61ee-a98b-4d49-87e9-6700bce75a1c\") "
Sep 14 13:29:19.077154 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:19.076960    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt9px\" (UniqueName: \"kubernetes.io/projected/d272c237-fddc-42d5-ad6f-c222c3370b96-kube-api-access-lt9px\") pod \"d272c237-fddc-42d5-ad6f-c222c3370b96\" (UID: \"d272c237-fddc-42d5-ad6f-c222c3370b96\") "
Sep 14 13:29:19.077154 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:19.076994    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1a61ee-a98b-4d49-87e9-6700bce75a1c-plugins-config-volume\") pod \"0d1a61ee-a98b-4d49-87e9-6700bce75a1c\" (UID: \"0d1a61ee-a98b-4d49-87e9-6700bce75a1c\") "
Sep 14 13:29:19.077154 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:19.077016    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d272c237-fddc-42d5-ad6f-c222c3370b96-plugins-config-volume\") pod \"d272c237-fddc-42d5-ad6f-c222c3370b96\" (UID: \"d272c237-fddc-42d5-ad6f-c222c3370b96\") "
Sep 14 13:29:19.077354 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:19.077330    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d1a61ee-a98b-4d49-87e9-6700bce75a1c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0d1a61ee-a98b-4d49-87e9-6700bce75a1c" (UID: "0d1a61ee-a98b-4d49-87e9-6700bce75a1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:29:19.077453 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:19.077432    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d272c237-fddc-42d5-ad6f-c222c3370b96-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d272c237-fddc-42d5-ad6f-c222c3370b96" (UID: "d272c237-fddc-42d5-ad6f-c222c3370b96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:29:19.079129 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:19.079108    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d272c237-fddc-42d5-ad6f-c222c3370b96-kube-api-access-lt9px" (OuterVolumeSpecName: "kube-api-access-lt9px") pod "d272c237-fddc-42d5-ad6f-c222c3370b96" (UID: "d272c237-fddc-42d5-ad6f-c222c3370b96"). InnerVolumeSpecName "kube-api-access-lt9px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:29:19.079211 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:19.079160    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d1a61ee-a98b-4d49-87e9-6700bce75a1c-kube-api-access-d5h8x" (OuterVolumeSpecName: "kube-api-access-d5h8x") pod "0d1a61ee-a98b-4d49-87e9-6700bce75a1c" (UID: "0d1a61ee-a98b-4d49-87e9-6700bce75a1c"). InnerVolumeSpecName "kube-api-access-d5h8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:29:19.178117 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:19.178024    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5h8x\" (UniqueName: \"kubernetes.io/projected/0d1a61ee-a98b-4d49-87e9-6700bce75a1c-kube-api-access-d5h8x\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:29:19.178117 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:19.178057    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt9px\" (UniqueName: \"kubernetes.io/projected/d272c237-fddc-42d5-ad6f-c222c3370b96-kube-api-access-lt9px\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:29:19.178117 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:19.178068    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1a61ee-a98b-4d49-87e9-6700bce75a1c-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:29:19.178117 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:19.178078    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d272c237-fddc-42d5-ad6f-c222c3370b96-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:29:19.869247 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:19.869160    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2"
Sep 14 13:29:19.869247 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:19.869190    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9"
Sep 14 13:29:19.869247 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:19.869181    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2" event={"ID":"d272c237-fddc-42d5-ad6f-c222c3370b96","Type":"ContainerDied","Data":"e7476c7d1ead3e806eeb2b445d28f9419240bd53135dc2449d6a2bb836bf0f82"}
Sep 14 13:29:19.869247 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:19.869249    2557 scope.go:117] "RemoveContainer" containerID="2c4d3c53bf32efe1c18bd7a7facfb588a47aeb7fc512bc999cbbcbe447e73b51"
Sep 14 13:29:19.893926 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:19.893901    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9"]
Sep 14 13:29:19.897585 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:19.897558    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-31063a-7f465569c-j5dt9"]
Sep 14 13:29:19.908211 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:19.908192    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2"]
Sep 14 13:29:19.914154 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:19.914131    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-31063a-84bc48cf5d-g6mh2"]
Sep 14 13:29:20.048898 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:20.048860    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d1a61ee-a98b-4d49-87e9-6700bce75a1c" path="/var/lib/kubelet/pods/0d1a61ee-a98b-4d49-87e9-6700bce75a1c/volumes"
Sep 14 13:29:20.049211 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:20.049196    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53c93d64-afc7-4a87-a13e-14eee62ed936" path="/var/lib/kubelet/pods/53c93d64-afc7-4a87-a13e-14eee62ed936/volumes"
Sep 14 13:29:20.049511 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:20.049497    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d272c237-fddc-42d5-ad6f-c222c3370b96" path="/var/lib/kubelet/pods/d272c237-fddc-42d5-ad6f-c222c3370b96/volumes"
Sep 14 13:29:21.608715 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.608686    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"]
Sep 14 13:29:21.609100 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.609069    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53c93d64-afc7-4a87-a13e-14eee62ed936" containerName="revoke-keys"
Sep 14 13:29:21.609100 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.609082    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="53c93d64-afc7-4a87-a13e-14eee62ed936" containerName="revoke-keys"
Sep 14 13:29:21.609100 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.609092    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e7b8e99-ce68-4bac-9a90-10dbfc580209" containerName="istio-proxy"
Sep 14 13:29:21.609100 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.609097    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e7b8e99-ce68-4bac-9a90-10dbfc580209" containerName="istio-proxy"
Sep 14 13:29:21.609287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.609111    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d272c237-fddc-42d5-ad6f-c222c3370b96" containerName="payload-pre-processing"
Sep 14 13:29:21.609287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.609117    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d272c237-fddc-42d5-ad6f-c222c3370b96" containerName="payload-pre-processing"
Sep 14 13:29:21.609287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.609126    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d1a61ee-a98b-4d49-87e9-6700bce75a1c" containerName="payload-processing"
Sep 14 13:29:21.609287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.609132    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d1a61ee-a98b-4d49-87e9-6700bce75a1c" containerName="payload-processing"
Sep 14 13:29:21.609287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.609204    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="53c93d64-afc7-4a87-a13e-14eee62ed936" containerName="revoke-keys"
Sep 14 13:29:21.609287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.609213    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e7b8e99-ce68-4bac-9a90-10dbfc580209" containerName="istio-proxy"
Sep 14 13:29:21.609287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.609221    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d272c237-fddc-42d5-ad6f-c222c3370b96" containerName="payload-pre-processing"
Sep 14 13:29:21.609287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.609229    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d1a61ee-a98b-4d49-87e9-6700bce75a1c" containerName="payload-processing"
Sep 14 13:29:21.613585 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.613563    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:21.616714 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.616693    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-14f3af\"/\"openshift-service-ca.crt\""
Sep 14 13:29:21.616900 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.616880    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-14f3af\"/\"kube-root-ca.crt\""
Sep 14 13:29:21.618134 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.618112    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-14f3af\"/\"default-dockercfg-pbl7q\""
Sep 14 13:29:21.618264 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.618137    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-14f3af\"/\"rate-test-model-14f3af-kserve-self-signed-certs\""
Sep 14 13:29:21.624448 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.624425    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"]
Sep 14 13:29:21.699258 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.699224    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-model-cache\") pod \"rate-test-model-14f3af-kserve-649fbc77f-5jm84\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") " pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:21.699258 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.699257    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk4vq\" (UniqueName: \"kubernetes.io/projected/d992158c-2000-4b70-bbbd-cb28149c60a0-kube-api-access-pk4vq\") pod \"rate-test-model-14f3af-kserve-649fbc77f-5jm84\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") " pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:21.699462 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.699279    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d992158c-2000-4b70-bbbd-cb28149c60a0-tls-certs\") pod \"rate-test-model-14f3af-kserve-649fbc77f-5jm84\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") " pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:21.699462 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.699355    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-tmp-dir\") pod \"rate-test-model-14f3af-kserve-649fbc77f-5jm84\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") " pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:21.699462 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.699407    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-home\") pod \"rate-test-model-14f3af-kserve-649fbc77f-5jm84\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") " pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:21.699565 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.699467    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-dshm\") pod \"rate-test-model-14f3af-kserve-649fbc77f-5jm84\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") " pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:21.799910 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.799877    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-home\") pod \"rate-test-model-14f3af-kserve-649fbc77f-5jm84\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") " pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:21.800110 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.799927    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-dshm\") pod \"rate-test-model-14f3af-kserve-649fbc77f-5jm84\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") " pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:21.800110 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.799956    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-model-cache\") pod \"rate-test-model-14f3af-kserve-649fbc77f-5jm84\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") " pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:21.800110 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.799983    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk4vq\" (UniqueName: \"kubernetes.io/projected/d992158c-2000-4b70-bbbd-cb28149c60a0-kube-api-access-pk4vq\") pod \"rate-test-model-14f3af-kserve-649fbc77f-5jm84\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") " pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:21.800110 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.800001    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d992158c-2000-4b70-bbbd-cb28149c60a0-tls-certs\") pod \"rate-test-model-14f3af-kserve-649fbc77f-5jm84\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") " pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:21.800110 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.800045    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-tmp-dir\") pod \"rate-test-model-14f3af-kserve-649fbc77f-5jm84\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") " pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:21.800385 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.800316    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-home\") pod \"rate-test-model-14f3af-kserve-649fbc77f-5jm84\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") " pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:21.800385 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.800357    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-model-cache\") pod \"rate-test-model-14f3af-kserve-649fbc77f-5jm84\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") " pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:21.800464 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.800401    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-tmp-dir\") pod \"rate-test-model-14f3af-kserve-649fbc77f-5jm84\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") " pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:21.802343 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.802321    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-dshm\") pod \"rate-test-model-14f3af-kserve-649fbc77f-5jm84\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") " pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:21.802455 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.802436    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d992158c-2000-4b70-bbbd-cb28149c60a0-tls-certs\") pod \"rate-test-model-14f3af-kserve-649fbc77f-5jm84\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") " pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:21.807394 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.807373    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk4vq\" (UniqueName: \"kubernetes.io/projected/d992158c-2000-4b70-bbbd-cb28149c60a0-kube-api-access-pk4vq\") pod \"rate-test-model-14f3af-kserve-649fbc77f-5jm84\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") " pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:21.923336 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:21.923244    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:22.208982 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:22.208952    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"]
Sep 14 13:29:22.212854 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:29:22.212818    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd992158c_2000_4b70_bbbd_cb28149c60a0.slice/crio-6f920c8f7158a10190ff043c9d0d8645b4e524c7edfce5d0bd6ce4e584b914b0 WatchSource:0}: Error finding container 6f920c8f7158a10190ff043c9d0d8645b4e524c7edfce5d0bd6ce4e584b914b0: Status 404 returned error can't find the container with id 6f920c8f7158a10190ff043c9d0d8645b4e524c7edfce5d0bd6ce4e584b914b0
Sep 14 13:29:22.886580 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:22.886543    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84" event={"ID":"d992158c-2000-4b70-bbbd-cb28149c60a0","Type":"ContainerStarted","Data":"a736d7a768a694f9cf5bde0fb7b74078795a8d8d491f0be635513ee39379efd0"}
Sep 14 13:29:22.886580 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:22.886584    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84" event={"ID":"d992158c-2000-4b70-bbbd-cb28149c60a0","Type":"ContainerStarted","Data":"6f920c8f7158a10190ff043c9d0d8645b4e524c7edfce5d0bd6ce4e584b914b0"}
Sep 14 13:29:22.909615 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:22.909560    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84" podStartSLOduration=1.909544216 podStartE2EDuration="1.909544216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:29:22.907887033 +0000 UTC m=+1881.342562552" watchObservedRunningTime="2026-09-14 13:29:22.909544216 +0000 UTC m=+1881.344219734"
Sep 14 13:29:26.319362 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:26.319287    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b8be88-5zv9j"]
Sep 14 13:29:26.319769 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:26.319751    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e7b8e99-ce68-4bac-9a90-10dbfc580209" containerName="istio-proxy"
Sep 14 13:29:26.319828 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:26.319772    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e7b8e99-ce68-4bac-9a90-10dbfc580209" containerName="istio-proxy"
Sep 14 13:29:26.319878 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:26.319839    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e7b8e99-ce68-4bac-9a90-10dbfc580209" containerName="istio-proxy"
Sep 14 13:29:26.322805 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:26.322785    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b8be88-5zv9j"
Sep 14 13:29:26.325585 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:26.325564    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbzp\""
Sep 14 13:29:26.331297 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:26.331273    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b8be88-5zv9j"]
Sep 14 13:29:26.342539 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:26.342511    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0823cf0-1f85-4c4c-a8ae-af5598a649f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b8be88-5zv9j\" (UID: \"c0823cf0-1f85-4c4c-a8ae-af5598a649f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b8be88-5zv9j"
Sep 14 13:29:26.443004 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:26.442966    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0823cf0-1f85-4c4c-a8ae-af5598a649f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b8be88-5zv9j\" (UID: \"c0823cf0-1f85-4c4c-a8ae-af5598a649f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b8be88-5zv9j"
Sep 14 13:29:26.443641 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:26.443621    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0823cf0-1f85-4c4c-a8ae-af5598a649f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b8be88-5zv9j\" (UID: \"c0823cf0-1f85-4c4c-a8ae-af5598a649f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b8be88-5zv9j"
Sep 14 13:29:26.635605 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:26.635531    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b8be88-5zv9j"
Sep 14 13:29:26.907034 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:26.906949    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b8be88-5zv9j"]
Sep 14 13:29:26.910024 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:29:26.909997    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0823cf0_1f85_4c4c_a8ae_af5598a649f5.slice/crio-11e2a525d84f3775df83bc3499822e24c11201491559197c815fa25728670cd4 WatchSource:0}: Error finding container 11e2a525d84f3775df83bc3499822e24c11201491559197c815fa25728670cd4: Status 404 returned error can't find the container with id 11e2a525d84f3775df83bc3499822e24c11201491559197c815fa25728670cd4
Sep 14 13:29:27.914668 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:27.914579    2557 generic.go:356] "Generic (PLEG): container finished" podID="c0823cf0-1f85-4c4c-a8ae-af5598a649f5" containerID="3f42c1f326b3fd593e6a0e0d10c610bf6c0c2cd503d57ba3ce643c4fecb4e05b" exitCode=0
Sep 14 13:29:27.915053 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:27.914664    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b8be88-5zv9j" event={"ID":"c0823cf0-1f85-4c4c-a8ae-af5598a649f5","Type":"ContainerDied","Data":"3f42c1f326b3fd593e6a0e0d10c610bf6c0c2cd503d57ba3ce643c4fecb4e05b"}
Sep 14 13:29:27.915053 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:27.914756    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b8be88-5zv9j" event={"ID":"c0823cf0-1f85-4c4c-a8ae-af5598a649f5","Type":"ContainerStarted","Data":"11e2a525d84f3775df83bc3499822e24c11201491559197c815fa25728670cd4"}
Sep 14 13:29:29.064154 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:29.064130    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b8be88-5zv9j"
Sep 14 13:29:29.173214 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:29.173128    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0823cf0-1f85-4c4c-a8ae-af5598a649f5-maas-api-service-ca\") pod \"c0823cf0-1f85-4c4c-a8ae-af5598a649f5\" (UID: \"c0823cf0-1f85-4c4c-a8ae-af5598a649f5\") "
Sep 14 13:29:29.173498 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:29.173476    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0823cf0-1f85-4c4c-a8ae-af5598a649f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c0823cf0-1f85-4c4c-a8ae-af5598a649f5" (UID: "c0823cf0-1f85-4c4c-a8ae-af5598a649f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:29:29.274591 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:29.274554    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0823cf0-1f85-4c4c-a8ae-af5598a649f5-maas-api-service-ca\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:29:29.926388 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:29.926355    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b8be88-5zv9j" event={"ID":"c0823cf0-1f85-4c4c-a8ae-af5598a649f5","Type":"ContainerDied","Data":"11e2a525d84f3775df83bc3499822e24c11201491559197c815fa25728670cd4"}
Sep 14 13:29:29.926388 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:29.926376    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b8be88-5zv9j"
Sep 14 13:29:29.926610 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:29.926405    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11e2a525d84f3775df83bc3499822e24c11201491559197c815fa25728670cd4"
Sep 14 13:29:31.923970 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:31.923931    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:31.924463 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:31.923991    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:31.937387 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:31.937361    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:31.948706 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:31.948677    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:29:37.222282 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:37.222249    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7"]
Sep 14 13:29:37.222804 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:37.222571    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7" podUID="1137a84f-0a7a-4a08-b43f-5d6273537a10" containerName="payload-pre-processing" containerID="cri-o://7fa8c3e4d0cdaa888e1bf2b63be1cd5b9400899c0d8a4854fc72cc7572570145" gracePeriod=30
Sep 14 13:29:37.439402 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:37.439367    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b8be88-5zv9j"]
Sep 14 13:29:37.444845 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:37.444811    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b8be88-5zv9j"]
Sep 14 13:29:37.470098 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:37.470070    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7"
Sep 14 13:29:37.549874 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:37.549791    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flb9q\" (UniqueName: \"kubernetes.io/projected/1137a84f-0a7a-4a08-b43f-5d6273537a10-kube-api-access-flb9q\") pod \"1137a84f-0a7a-4a08-b43f-5d6273537a10\" (UID: \"1137a84f-0a7a-4a08-b43f-5d6273537a10\") "
Sep 14 13:29:37.549874 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:37.549854    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1137a84f-0a7a-4a08-b43f-5d6273537a10-plugins-config-volume\") pod \"1137a84f-0a7a-4a08-b43f-5d6273537a10\" (UID: \"1137a84f-0a7a-4a08-b43f-5d6273537a10\") "
Sep 14 13:29:37.550289 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:37.550259    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1137a84f-0a7a-4a08-b43f-5d6273537a10-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1137a84f-0a7a-4a08-b43f-5d6273537a10" (UID: "1137a84f-0a7a-4a08-b43f-5d6273537a10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:29:37.552119 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:37.552092    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1137a84f-0a7a-4a08-b43f-5d6273537a10-kube-api-access-flb9q" (OuterVolumeSpecName: "kube-api-access-flb9q") pod "1137a84f-0a7a-4a08-b43f-5d6273537a10" (UID: "1137a84f-0a7a-4a08-b43f-5d6273537a10"). InnerVolumeSpecName "kube-api-access-flb9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:29:37.650636 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:37.650593    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flb9q\" (UniqueName: \"kubernetes.io/projected/1137a84f-0a7a-4a08-b43f-5d6273537a10-kube-api-access-flb9q\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:29:37.650636 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:37.650635    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1137a84f-0a7a-4a08-b43f-5d6273537a10-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:29:37.972637 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:37.972598    2557 generic.go:356] "Generic (PLEG): container finished" podID="1137a84f-0a7a-4a08-b43f-5d6273537a10" containerID="7fa8c3e4d0cdaa888e1bf2b63be1cd5b9400899c0d8a4854fc72cc7572570145" exitCode=0
Sep 14 13:29:37.972864 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:37.972649    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7"
Sep 14 13:29:37.972864 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:37.972653    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7" event={"ID":"1137a84f-0a7a-4a08-b43f-5d6273537a10","Type":"ContainerDied","Data":"7fa8c3e4d0cdaa888e1bf2b63be1cd5b9400899c0d8a4854fc72cc7572570145"}
Sep 14 13:29:37.972864 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:37.972802    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7" event={"ID":"1137a84f-0a7a-4a08-b43f-5d6273537a10","Type":"ContainerDied","Data":"af3bbe78625495e3d8254169d976533f435b31116d5bf2d53053639642f65236"}
Sep 14 13:29:37.972864 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:37.972831    2557 scope.go:117] "RemoveContainer" containerID="7fa8c3e4d0cdaa888e1bf2b63be1cd5b9400899c0d8a4854fc72cc7572570145"
Sep 14 13:29:37.982539 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:37.982521    2557 scope.go:117] "RemoveContainer" containerID="7fa8c3e4d0cdaa888e1bf2b63be1cd5b9400899c0d8a4854fc72cc7572570145"
Sep 14 13:29:37.982867 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:29:37.982850    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa8c3e4d0cdaa888e1bf2b63be1cd5b9400899c0d8a4854fc72cc7572570145\": container with ID starting with 7fa8c3e4d0cdaa888e1bf2b63be1cd5b9400899c0d8a4854fc72cc7572570145 not found: ID does not exist" containerID="7fa8c3e4d0cdaa888e1bf2b63be1cd5b9400899c0d8a4854fc72cc7572570145"
Sep 14 13:29:37.982923 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:37.982906    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa8c3e4d0cdaa888e1bf2b63be1cd5b9400899c0d8a4854fc72cc7572570145"} err="failed to get container status \"7fa8c3e4d0cdaa888e1bf2b63be1cd5b9400899c0d8a4854fc72cc7572570145\": rpc error: code = NotFound desc = could not find container \"7fa8c3e4d0cdaa888e1bf2b63be1cd5b9400899c0d8a4854fc72cc7572570145\": container with ID starting with 7fa8c3e4d0cdaa888e1bf2b63be1cd5b9400899c0d8a4854fc72cc7572570145 not found: ID does not exist"
Sep 14 13:29:37.999090 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:37.999063    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7"]
Sep 14 13:29:38.010188 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:38.010163    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8be88-5d9bbf6b98-6m2j7"]
Sep 14 13:29:38.054896 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:38.054862    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1137a84f-0a7a-4a08-b43f-5d6273537a10" path="/var/lib/kubelet/pods/1137a84f-0a7a-4a08-b43f-5d6273537a10/volumes"
Sep 14 13:29:38.055179 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:29:38.055166    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0823cf0-1f85-4c4c-a8ae-af5598a649f5" path="/var/lib/kubelet/pods/c0823cf0-1f85-4c4c-a8ae-af5598a649f5/volumes"
Sep 14 13:30:00.141995 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:00.141963    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14f3af-29823210-xbdng"]
Sep 14 13:30:00.142460 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:00.142319    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0823cf0-1f85-4c4c-a8ae-af5598a649f5" containerName="revoke-keys"
Sep 14 13:30:00.142460 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:00.142330    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0823cf0-1f85-4c4c-a8ae-af5598a649f5" containerName="revoke-keys"
Sep 14 13:30:00.142460 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:00.142347    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1137a84f-0a7a-4a08-b43f-5d6273537a10" containerName="payload-pre-processing"
Sep 14 13:30:00.142460 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:00.142353    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1137a84f-0a7a-4a08-b43f-5d6273537a10" containerName="payload-pre-processing"
Sep 14 13:30:00.142460 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:00.142411    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1137a84f-0a7a-4a08-b43f-5d6273537a10" containerName="payload-pre-processing"
Sep 14 13:30:00.142460 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:00.142420    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0823cf0-1f85-4c4c-a8ae-af5598a649f5" containerName="revoke-keys"
Sep 14 13:30:00.146428 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:00.146405    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823210-qskrr"]
Sep 14 13:30:00.146556 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:00.146515    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14f3af-29823210-xbdng"
Sep 14 13:30:00.149336 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:00.149315    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbzp\""
Sep 14 13:30:00.149801 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:00.149780    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823210-qskrr"
Sep 14 13:30:00.153478 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:00.153420    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14f3af-29823210-xbdng"]
Sep 14 13:30:00.157554 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:00.157535    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823210-qskrr"]
Sep 14 13:30:00.160167 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:00.160150    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14f3af-29823210-xbdng"
Sep 14 13:30:00.168259 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:00.168239    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823210-qskrr"
Sep 14 13:30:00.439033 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:00.439010    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14f3af-29823210-xbdng"]
Sep 14 13:30:00.442120 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:30:00.442093    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe008d5d_4051_44c1_8616_26fa3524bef0.slice/crio-83949b40fb97abd4e5aba9f22235ce845bbcca631570c4c44aaf7cb5caf1ce96 WatchSource:0}: Error finding container 83949b40fb97abd4e5aba9f22235ce845bbcca631570c4c44aaf7cb5caf1ce96: Status 404 returned error can't find the container with id 83949b40fb97abd4e5aba9f22235ce845bbcca631570c4c44aaf7cb5caf1ce96
Sep 14 13:30:00.449497 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:00.449472    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823210-qskrr"]
Sep 14 13:30:00.451389 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:30:00.451364    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175254a6_1f95_48d7_a5b2_2cde0508afa8.slice/crio-37976a4189f6b2e1633b2dcf7a821f766063923bb4f006963f7235174ccc4c9f WatchSource:0}: Error finding container 37976a4189f6b2e1633b2dcf7a821f766063923bb4f006963f7235174ccc4c9f: Status 404 returned error can't find the container with id 37976a4189f6b2e1633b2dcf7a821f766063923bb4f006963f7235174ccc4c9f
Sep 14 13:30:01.093343 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:01.093242    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14f3af-29823210-xbdng" event={"ID":"be008d5d-4051-44c1-8616-26fa3524bef0","Type":"ContainerStarted","Data":"5d2d7cde776ba5f0c74a90a0d39fdb36b76426b076cf0adfc6a8653500864f20"}
Sep 14 13:30:01.093343 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:01.093301    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14f3af-29823210-xbdng" event={"ID":"be008d5d-4051-44c1-8616-26fa3524bef0","Type":"ContainerStarted","Data":"83949b40fb97abd4e5aba9f22235ce845bbcca631570c4c44aaf7cb5caf1ce96"}
Sep 14 13:30:01.094828 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:01.094799    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823210-qskrr" event={"ID":"175254a6-1f95-48d7-a5b2-2cde0508afa8","Type":"ContainerStarted","Data":"309a8c725592029a6c0f8fbcd5347db8d8962dc68ad57c37095832036d9f6bd6"}
Sep 14 13:30:01.094949 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:01.094853    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823210-qskrr" event={"ID":"175254a6-1f95-48d7-a5b2-2cde0508afa8","Type":"ContainerStarted","Data":"37976a4189f6b2e1633b2dcf7a821f766063923bb4f006963f7235174ccc4c9f"}
Sep 14 13:30:01.110870 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:01.110826    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14f3af-29823210-xbdng" podStartSLOduration=1.1108125150000001 podStartE2EDuration="1.110812515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:30:01.109505375 +0000 UTC m=+1919.544180894" watchObservedRunningTime="2026-09-14 13:30:01.110812515 +0000 UTC m=+1919.545488033"
Sep 14 13:30:01.124392 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:01.124352    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823210-qskrr" podStartSLOduration=1.124337935 podStartE2EDuration="1.124337935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:30:01.12354306 +0000 UTC m=+1919.558218583" watchObservedRunningTime="2026-09-14 13:30:01.124337935 +0000 UTC m=+1919.559013454"
Sep 14 13:30:02.102047 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:02.102015    2557 generic.go:356] "Generic (PLEG): container finished" podID="175254a6-1f95-48d7-a5b2-2cde0508afa8" containerID="309a8c725592029a6c0f8fbcd5347db8d8962dc68ad57c37095832036d9f6bd6" exitCode=0
Sep 14 13:30:02.102456 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:02.102095    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823210-qskrr" event={"ID":"175254a6-1f95-48d7-a5b2-2cde0508afa8","Type":"ContainerDied","Data":"309a8c725592029a6c0f8fbcd5347db8d8962dc68ad57c37095832036d9f6bd6"}
Sep 14 13:30:02.103726 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:02.103703    2557 generic.go:356] "Generic (PLEG): container finished" podID="be008d5d-4051-44c1-8616-26fa3524bef0" containerID="5d2d7cde776ba5f0c74a90a0d39fdb36b76426b076cf0adfc6a8653500864f20" exitCode=0
Sep 14 13:30:02.103834 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:02.103773    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14f3af-29823210-xbdng" event={"ID":"be008d5d-4051-44c1-8616-26fa3524bef0","Type":"ContainerDied","Data":"5d2d7cde776ba5f0c74a90a0d39fdb36b76426b076cf0adfc6a8653500864f20"}
Sep 14 13:30:02.565960 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:02.565932    2557 scope.go:117] "RemoveContainer" containerID="afea69578344fff65dfb420f615e89c17ef79f7d6b13330df2a476bce8af1870"
Sep 14 13:30:03.248973 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:03.248944    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14f3af-29823210-xbdng"
Sep 14 13:30:03.275022 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:03.274994    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823210-qskrr"
Sep 14 13:30:04.114883 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:04.114853    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14f3af-29823210-xbdng"
Sep 14 13:30:04.114883 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:04.114878    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14f3af-29823210-xbdng" event={"ID":"be008d5d-4051-44c1-8616-26fa3524bef0","Type":"ContainerDied","Data":"83949b40fb97abd4e5aba9f22235ce845bbcca631570c4c44aaf7cb5caf1ce96"}
Sep 14 13:30:04.115186 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:04.114923    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83949b40fb97abd4e5aba9f22235ce845bbcca631570c4c44aaf7cb5caf1ce96"
Sep 14 13:30:04.116249 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:04.116231    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823210-qskrr"
Sep 14 13:30:04.116368 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:04.116263    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823210-qskrr" event={"ID":"175254a6-1f95-48d7-a5b2-2cde0508afa8","Type":"ContainerDied","Data":"37976a4189f6b2e1633b2dcf7a821f766063923bb4f006963f7235174ccc4c9f"}
Sep 14 13:30:04.116368 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:04.116295    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37976a4189f6b2e1633b2dcf7a821f766063923bb4f006963f7235174ccc4c9f"
Sep 14 13:30:18.539603 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:18.539571    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"]
Sep 14 13:30:20.248577 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.248548    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"]
Sep 14 13:30:20.248967 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.248930    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="175254a6-1f95-48d7-a5b2-2cde0508afa8" containerName="cleanup"
Sep 14 13:30:20.248967 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.248942    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="175254a6-1f95-48d7-a5b2-2cde0508afa8" containerName="cleanup"
Sep 14 13:30:20.248967 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.248956    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be008d5d-4051-44c1-8616-26fa3524bef0" containerName="cleanup"
Sep 14 13:30:20.248967 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.248962    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="be008d5d-4051-44c1-8616-26fa3524bef0" containerName="cleanup"
Sep 14 13:30:20.249098 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.249025    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="175254a6-1f95-48d7-a5b2-2cde0508afa8" containerName="cleanup"
Sep 14 13:30:20.249098 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.249035    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="be008d5d-4051-44c1-8616-26fa3524bef0" containerName="cleanup"
Sep 14 13:30:20.252035 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.252020    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.256841 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.256817    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c87a95-openshift-default-dockercfg-pqh4l\""
Sep 14 13:30:20.266098 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.266074    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"]
Sep 14 13:30:20.326281 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.326251    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-credential-socket\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.326444 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.326290    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-token\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.326444 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.326325    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-workload-socket\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.326444 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.326432    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-data\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.326579 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.326470    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ab6ae2-e510-4102-8efc-40cb5c386af3-istiod-ca-cert\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.326579 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.326531    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-workload-certs\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.326678 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.326576    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9sfl\" (UniqueName: \"kubernetes.io/projected/17ab6ae2-e510-4102-8efc-40cb5c386af3-kube-api-access-q9sfl\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.326776 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.326699    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-envoy\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.326776 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.326730    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-podinfo\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.433979 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.427933    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-credential-socket\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.433979 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.427981    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-token\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.433979 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.428031    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-workload-socket\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.433979 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.428100    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-data\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.433979 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.428134    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ab6ae2-e510-4102-8efc-40cb5c386af3-istiod-ca-cert\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.433979 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.428178    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-workload-certs\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.433979 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.428226    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9sfl\" (UniqueName: \"kubernetes.io/projected/17ab6ae2-e510-4102-8efc-40cb5c386af3-kube-api-access-q9sfl\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.433979 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.428295    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-envoy\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.433979 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.428324    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-podinfo\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.433979 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.429569    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-credential-socket\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.433979 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.429856    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-workload-certs\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.433979 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.430032    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ab6ae2-e510-4102-8efc-40cb5c386af3-istiod-ca-cert\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.433979 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.430565    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-workload-socket\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.433979 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.430819    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-data\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.434872 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.434134    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-envoy\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.435686 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.435667    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-podinfo\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.438594 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.438574    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-token\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.439208 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.439187    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9sfl\" (UniqueName: \"kubernetes.io/projected/17ab6ae2-e510-4102-8efc-40cb5c386af3-kube-api-access-q9sfl\") pod \"e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") " pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.562636 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.562554    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:20.840550 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.840521    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"]
Sep 14 13:30:20.842001 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:30:20.841975    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ab6ae2_e510_4102_8efc_40cb5c386af3.slice/crio-677f1bf0903986946d684108a75f4dddf7a3e8e7fe2d06b708568957f317d412 WatchSource:0}: Error finding container 677f1bf0903986946d684108a75f4dddf7a3e8e7fe2d06b708568957f317d412: Status 404 returned error can't find the container with id 677f1bf0903986946d684108a75f4dddf7a3e8e7fe2d06b708568957f317d412
Sep 14 13:30:20.844218 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.844186    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:30:20.844292 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.844251    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:30:20.844292 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:20.844279    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:30:21.213755 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:21.213709    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k" event={"ID":"17ab6ae2-e510-4102-8efc-40cb5c386af3","Type":"ContainerStarted","Data":"ae2cae829194cdc9bc0f7b7574392058a9ae9bae44dd560cf6f50798f148c6b4"}
Sep 14 13:30:21.213940 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:21.213778    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k" event={"ID":"17ab6ae2-e510-4102-8efc-40cb5c386af3","Type":"ContainerStarted","Data":"677f1bf0903986946d684108a75f4dddf7a3e8e7fe2d06b708568957f317d412"}
Sep 14 13:30:21.237794 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:21.237709    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k" podStartSLOduration=1.23769111 podStartE2EDuration="1.23769111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:30:21.237518089 +0000 UTC m=+1939.672193608" watchObservedRunningTime="2026-09-14 13:30:21.23769111 +0000 UTC m=+1939.672366632"
Sep 14 13:30:21.562960 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:21.562877    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:21.568317 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:21.568290    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:22.218795 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:22.218767    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:22.219769 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:22.219730    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:25.600643 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:25.600605    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584"]
Sep 14 13:30:25.604229 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:25.604212    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584"
Sep 14 13:30:25.607222 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:25.607202    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c87a95-dockercfg-q8494\""
Sep 14 13:30:25.607358 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:25.607244    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c87a95\""
Sep 14 13:30:25.616263 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:25.616237    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584"]
Sep 14 13:30:25.679180 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:25.679152    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v28r\" (UniqueName: \"kubernetes.io/projected/74290e42-38b3-4436-a904-3c2960330824-kube-api-access-2v28r\") pod \"payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584\" (UID: \"74290e42-38b3-4436-a904-3c2960330824\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584"
Sep 14 13:30:25.679345 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:25.679197    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74290e42-38b3-4436-a904-3c2960330824-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584\" (UID: \"74290e42-38b3-4436-a904-3c2960330824\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584"
Sep 14 13:30:25.780783 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:25.780718    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v28r\" (UniqueName: \"kubernetes.io/projected/74290e42-38b3-4436-a904-3c2960330824-kube-api-access-2v28r\") pod \"payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584\" (UID: \"74290e42-38b3-4436-a904-3c2960330824\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584"
Sep 14 13:30:25.780957 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:25.780803    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74290e42-38b3-4436-a904-3c2960330824-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584\" (UID: \"74290e42-38b3-4436-a904-3c2960330824\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584"
Sep 14 13:30:25.781544 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:25.781526    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74290e42-38b3-4436-a904-3c2960330824-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584\" (UID: \"74290e42-38b3-4436-a904-3c2960330824\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584"
Sep 14 13:30:25.791214 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:25.791186    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v28r\" (UniqueName: \"kubernetes.io/projected/74290e42-38b3-4436-a904-3c2960330824-kube-api-access-2v28r\") pod \"payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584\" (UID: \"74290e42-38b3-4436-a904-3c2960330824\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584"
Sep 14 13:30:25.915553 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:25.915467    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584"
Sep 14 13:30:26.187858 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:26.187835    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584"]
Sep 14 13:30:26.190121 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:30:26.190091    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74290e42_38b3_4436_a904_3c2960330824.slice/crio-a9617940c8b3abd86f570b5b7c411c171201982bdc7a7eabadb0788d5aa16a3f WatchSource:0}: Error finding container a9617940c8b3abd86f570b5b7c411c171201982bdc7a7eabadb0788d5aa16a3f: Status 404 returned error can't find the container with id a9617940c8b3abd86f570b5b7c411c171201982bdc7a7eabadb0788d5aa16a3f
Sep 14 13:30:26.242715 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:26.242690    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584" event={"ID":"74290e42-38b3-4436-a904-3c2960330824","Type":"ContainerStarted","Data":"a9617940c8b3abd86f570b5b7c411c171201982bdc7a7eabadb0788d5aa16a3f"}
Sep 14 13:30:27.249195 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:27.249160    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584" event={"ID":"74290e42-38b3-4436-a904-3c2960330824","Type":"ContainerStarted","Data":"27bca0bf8e49f41136dd2ab3faac4abdedeec1a28ac5b7d2b8081f0db7cff12a"}
Sep 14 13:30:27.249667 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:27.249227    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584"
Sep 14 13:30:27.268196 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:27.268149    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584" podStartSLOduration=2.268134056 podStartE2EDuration="2.268134056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:30:27.266568685 +0000 UTC m=+1945.701244206" watchObservedRunningTime="2026-09-14 13:30:27.268134056 +0000 UTC m=+1945.702809581"
Sep 14 13:30:33.565274 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:33.565207    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84" podUID="d992158c-2000-4b70-bbbd-cb28149c60a0" containerName="main" containerID="cri-o://a736d7a768a694f9cf5bde0fb7b74078795a8d8d491f0be635513ee39379efd0" gracePeriod=45
Sep 14 13:30:33.879849 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:33.879825    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:30:33.966765 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:33.960038    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-model-cache\") pod \"d992158c-2000-4b70-bbbd-cb28149c60a0\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") "
Sep 14 13:30:33.966765 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:33.960104    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-tmp-dir\") pod \"d992158c-2000-4b70-bbbd-cb28149c60a0\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") "
Sep 14 13:30:33.966765 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:33.960138    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d992158c-2000-4b70-bbbd-cb28149c60a0-tls-certs\") pod \"d992158c-2000-4b70-bbbd-cb28149c60a0\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") "
Sep 14 13:30:33.966765 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:33.960214    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-dshm\") pod \"d992158c-2000-4b70-bbbd-cb28149c60a0\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") "
Sep 14 13:30:33.966765 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:33.960302    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pk4vq\" (UniqueName: \"kubernetes.io/projected/d992158c-2000-4b70-bbbd-cb28149c60a0-kube-api-access-pk4vq\") pod \"d992158c-2000-4b70-bbbd-cb28149c60a0\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") "
Sep 14 13:30:33.966765 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:33.960341    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-home\") pod \"d992158c-2000-4b70-bbbd-cb28149c60a0\" (UID: \"d992158c-2000-4b70-bbbd-cb28149c60a0\") "
Sep 14 13:30:33.971828 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:33.971337    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d992158c-2000-4b70-bbbd-cb28149c60a0" (UID: "d992158c-2000-4b70-bbbd-cb28149c60a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:30:33.971828 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:33.971513    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "d992158c-2000-4b70-bbbd-cb28149c60a0" (UID: "d992158c-2000-4b70-bbbd-cb28149c60a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:30:33.971828 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:33.971658    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-home" (OuterVolumeSpecName: "home") pod "d992158c-2000-4b70-bbbd-cb28149c60a0" (UID: "d992158c-2000-4b70-bbbd-cb28149c60a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:30:33.980555 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:33.979201    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d992158c-2000-4b70-bbbd-cb28149c60a0-kube-api-access-pk4vq" (OuterVolumeSpecName: "kube-api-access-pk4vq") pod "d992158c-2000-4b70-bbbd-cb28149c60a0" (UID: "d992158c-2000-4b70-bbbd-cb28149c60a0"). InnerVolumeSpecName "kube-api-access-pk4vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:30:33.980555 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:33.979405    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-dshm" (OuterVolumeSpecName: "dshm") pod "d992158c-2000-4b70-bbbd-cb28149c60a0" (UID: "d992158c-2000-4b70-bbbd-cb28149c60a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:30:33.980958 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:33.980928    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d992158c-2000-4b70-bbbd-cb28149c60a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d992158c-2000-4b70-bbbd-cb28149c60a0" (UID: "d992158c-2000-4b70-bbbd-cb28149c60a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:30:34.070633 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:34.070482    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-dshm\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:30:34.070633 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:34.070550    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk4vq\" (UniqueName: \"kubernetes.io/projected/d992158c-2000-4b70-bbbd-cb28149c60a0-kube-api-access-pk4vq\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:30:34.070633 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:34.070568    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-home\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:30:34.070633 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:34.070582    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-model-cache\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:30:34.070633 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:34.070595    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d992158c-2000-4b70-bbbd-cb28149c60a0-tmp-dir\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:30:34.070633 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:34.070608    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d992158c-2000-4b70-bbbd-cb28149c60a0-tls-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:30:34.290242 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:34.290211    2557 generic.go:356] "Generic (PLEG): container finished" podID="d992158c-2000-4b70-bbbd-cb28149c60a0" containerID="a736d7a768a694f9cf5bde0fb7b74078795a8d8d491f0be635513ee39379efd0" exitCode=0
Sep 14 13:30:34.290424 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:34.290264    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"
Sep 14 13:30:34.290424 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:34.290295    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84" event={"ID":"d992158c-2000-4b70-bbbd-cb28149c60a0","Type":"ContainerDied","Data":"a736d7a768a694f9cf5bde0fb7b74078795a8d8d491f0be635513ee39379efd0"}
Sep 14 13:30:34.290424 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:34.290341    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84" event={"ID":"d992158c-2000-4b70-bbbd-cb28149c60a0","Type":"ContainerDied","Data":"6f920c8f7158a10190ff043c9d0d8645b4e524c7edfce5d0bd6ce4e584b914b0"}
Sep 14 13:30:34.290424 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:34.290359    2557 scope.go:117] "RemoveContainer" containerID="a736d7a768a694f9cf5bde0fb7b74078795a8d8d491f0be635513ee39379efd0"
Sep 14 13:30:34.299084 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:34.299065    2557 scope.go:117] "RemoveContainer" containerID="a736d7a768a694f9cf5bde0fb7b74078795a8d8d491f0be635513ee39379efd0"
Sep 14 13:30:34.299350 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:30:34.299328    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a736d7a768a694f9cf5bde0fb7b74078795a8d8d491f0be635513ee39379efd0\": container with ID starting with a736d7a768a694f9cf5bde0fb7b74078795a8d8d491f0be635513ee39379efd0 not found: ID does not exist" containerID="a736d7a768a694f9cf5bde0fb7b74078795a8d8d491f0be635513ee39379efd0"
Sep 14 13:30:34.299432 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:34.299356    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a736d7a768a694f9cf5bde0fb7b74078795a8d8d491f0be635513ee39379efd0"} err="failed to get container status \"a736d7a768a694f9cf5bde0fb7b74078795a8d8d491f0be635513ee39379efd0\": rpc error: code = NotFound desc = could not find container \"a736d7a768a694f9cf5bde0fb7b74078795a8d8d491f0be635513ee39379efd0\": container with ID starting with a736d7a768a694f9cf5bde0fb7b74078795a8d8d491f0be635513ee39379efd0 not found: ID does not exist"
Sep 14 13:30:34.308228 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:34.308207    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"]
Sep 14 13:30:34.313837 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:34.313815    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-14f3af/rate-test-model-14f3af-kserve-649fbc77f-5jm84"]
Sep 14 13:30:36.048905 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:36.048864    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d992158c-2000-4b70-bbbd-cb28149c60a0" path="/var/lib/kubelet/pods/d992158c-2000-4b70-bbbd-cb28149c60a0/volumes"
Sep 14 13:30:38.258650 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:38.258617    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584"
Sep 14 13:30:40.332072 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:40.331983    2557 generic.go:356] "Generic (PLEG): container finished" podID="17ab6ae2-e510-4102-8efc-40cb5c386af3" containerID="ae2cae829194cdc9bc0f7b7574392058a9ae9bae44dd560cf6f50798f148c6b4" exitCode=0
Sep 14 13:30:40.332582 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:40.332062    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k" event={"ID":"17ab6ae2-e510-4102-8efc-40cb5c386af3","Type":"ContainerDied","Data":"ae2cae829194cdc9bc0f7b7574392058a9ae9bae44dd560cf6f50798f148c6b4"}
Sep 14 13:30:40.332582 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:40.332546    2557 scope.go:117] "RemoveContainer" containerID="ae2cae829194cdc9bc0f7b7574392058a9ae9bae44dd560cf6f50798f148c6b4"
Sep 14 13:30:40.333801 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:40.333771    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:30:40.333895 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:40.333838    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:30:40.333895 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:40.333869    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:30:41.339831 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:41.339789    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k" event={"ID":"17ab6ae2-e510-4102-8efc-40cb5c386af3","Type":"ContainerStarted","Data":"d41b8d039ae4ddc73d1c6f2dc48fb585a585845cfb935d7146cf0c72d873c7e2"}
Sep 14 13:30:41.563979 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:41.563927    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:41.564190 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:41.564072    2557 patch_prober.go:28] interesting pod/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Sep 14 13:30:41.564190 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:41.564147    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k" podUID="17ab6ae2-e510-4102-8efc-40cb5c386af3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Sep 14 13:30:42.567146 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:42.567111    2557 patch_prober.go:28] interesting pod/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Sep 14 13:30:42.567653 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:42.567178    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k" podUID="17ab6ae2-e510-4102-8efc-40cb5c386af3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Sep 14 13:30:43.563262 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:43.563226    2557 patch_prober.go:28] interesting pod/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Sep 14 13:30:43.563464 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:43.563291    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k" podUID="17ab6ae2-e510-4102-8efc-40cb5c386af3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Sep 14 13:30:44.563517 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:44.563480    2557 patch_prober.go:28] interesting pod/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Sep 14 13:30:44.564113 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:44.564083    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k" podUID="17ab6ae2-e510-4102-8efc-40cb5c386af3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Sep 14 13:30:45.563435 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:45.563395    2557 patch_prober.go:28] interesting pod/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Sep 14 13:30:45.563651 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:45.563466    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k" podUID="17ab6ae2-e510-4102-8efc-40cb5c386af3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Sep 14 13:30:46.567294 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:46.567262    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:46.567757 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:46.567540    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:46.568151 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:46.568136    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:30:49.044217 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.044184    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"]
Sep 14 13:30:49.044609 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.044568    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d992158c-2000-4b70-bbbd-cb28149c60a0" containerName="main"
Sep 14 13:30:49.044609 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.044579    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d992158c-2000-4b70-bbbd-cb28149c60a0" containerName="main"
Sep 14 13:30:49.044683 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.044643    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d992158c-2000-4b70-bbbd-cb28149c60a0" containerName="main"
Sep 14 13:30:49.047809 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.047793    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.051878 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.051857    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-27ae76-openshift-default-dockercfg-qpqlm\""
Sep 14 13:30:49.063826 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.063807    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"]
Sep 14 13:30:49.112090 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.112064    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-credential-socket\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.112253 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.112103    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-data\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.112253 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.112150    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-envoy\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.112253 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.112185    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd99aea0-8666-4eb5-9f98-3f371211f521-istiod-ca-cert\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.112253 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.112202    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-podinfo\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.112407 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.112315    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrw8f\" (UniqueName: \"kubernetes.io/projected/bd99aea0-8666-4eb5-9f98-3f371211f521-kube-api-access-mrw8f\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.112407 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.112379    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-token\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.112491 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.112410    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-workload-socket\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.112491 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.112427    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-workload-certs\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.213214 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.213180    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd99aea0-8666-4eb5-9f98-3f371211f521-istiod-ca-cert\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.213214 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.213215    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-podinfo\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.213425 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.213244    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrw8f\" (UniqueName: \"kubernetes.io/projected/bd99aea0-8666-4eb5-9f98-3f371211f521-kube-api-access-mrw8f\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.213425 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.213369    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-token\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.213425 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.213412    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-workload-socket\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.213589 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.213438    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-workload-certs\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.213589 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.213509    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-credential-socket\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.213589 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.213542    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-data\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.213787 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.213598    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-envoy\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.213876 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.213858    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-workload-socket\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.213944 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.213911    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-workload-certs\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.214003 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.213983    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-data\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.214049 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.214018    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-credential-socket\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.214085 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.214046    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd99aea0-8666-4eb5-9f98-3f371211f521-istiod-ca-cert\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.215766 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.215729    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-podinfo\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.215871 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.215768    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-envoy\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.222464 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.222441    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-token\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.223102 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.223080    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrw8f\" (UniqueName: \"kubernetes.io/projected/bd99aea0-8666-4eb5-9f98-3f371211f521-kube-api-access-mrw8f\") pod \"e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") " pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.358990 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.358906    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:49.648986 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.648962    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"]
Sep 14 13:30:49.651312 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:30:49.651272    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd99aea0_8666_4eb5_9f98_3f371211f521.slice/crio-6c4233fac60d5521dd53f3617f24c12dc9543c123ae34d13baaf1305cbd6496b WatchSource:0}: Error finding container 6c4233fac60d5521dd53f3617f24c12dc9543c123ae34d13baaf1305cbd6496b: Status 404 returned error can't find the container with id 6c4233fac60d5521dd53f3617f24c12dc9543c123ae34d13baaf1305cbd6496b
Sep 14 13:30:49.653430 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.653399    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:30:49.653510 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.653457    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:30:49.653510 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:49.653486    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:30:50.399415 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:50.399381    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q" event={"ID":"bd99aea0-8666-4eb5-9f98-3f371211f521","Type":"ContainerStarted","Data":"86cbb1507615d12b5ecb165ae2fc217b37a38a85237a48bad013d488a934ee81"}
Sep 14 13:30:50.399838 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:50.399444    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q" event={"ID":"bd99aea0-8666-4eb5-9f98-3f371211f521","Type":"ContainerStarted","Data":"6c4233fac60d5521dd53f3617f24c12dc9543c123ae34d13baaf1305cbd6496b"}
Sep 14 13:30:50.422417 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:50.422372    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q" podStartSLOduration=1.422357288 podStartE2EDuration="1.422357288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:30:50.420192145 +0000 UTC m=+1968.854867664" watchObservedRunningTime="2026-09-14 13:30:50.422357288 +0000 UTC m=+1968.857032807"
Sep 14 13:30:51.360090 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:51.360045    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:51.365727 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:51.365701    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:51.404363 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:51.404337    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:51.405181 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:51.405163    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:30:55.268851 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:55.268764    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t"]
Sep 14 13:30:55.272659 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:55.272635    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t"
Sep 14 13:30:55.275502 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:55.275480    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-27ae76\""
Sep 14 13:30:55.284908 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:55.284879    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t"]
Sep 14 13:30:55.370890 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:55.370864    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78fpv\" (UniqueName: \"kubernetes.io/projected/64007c09-5401-47c2-a0d4-853382b7a983-kube-api-access-78fpv\") pod \"payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t\" (UID: \"64007c09-5401-47c2-a0d4-853382b7a983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t"
Sep 14 13:30:55.371041 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:55.370995    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64007c09-5401-47c2-a0d4-853382b7a983-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t\" (UID: \"64007c09-5401-47c2-a0d4-853382b7a983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t"
Sep 14 13:30:55.472196 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:55.472162    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64007c09-5401-47c2-a0d4-853382b7a983-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t\" (UID: \"64007c09-5401-47c2-a0d4-853382b7a983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t"
Sep 14 13:30:55.472392 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:55.472209    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78fpv\" (UniqueName: \"kubernetes.io/projected/64007c09-5401-47c2-a0d4-853382b7a983-kube-api-access-78fpv\") pod \"payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t\" (UID: \"64007c09-5401-47c2-a0d4-853382b7a983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t"
Sep 14 13:30:55.472943 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:55.472917    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64007c09-5401-47c2-a0d4-853382b7a983-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t\" (UID: \"64007c09-5401-47c2-a0d4-853382b7a983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t"
Sep 14 13:30:55.481350 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:55.481330    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78fpv\" (UniqueName: \"kubernetes.io/projected/64007c09-5401-47c2-a0d4-853382b7a983-kube-api-access-78fpv\") pod \"payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t\" (UID: \"64007c09-5401-47c2-a0d4-853382b7a983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t"
Sep 14 13:30:55.583887 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:55.583812    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t"
Sep 14 13:30:55.853477 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:55.853454    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t"]
Sep 14 13:30:55.855627 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:30:55.855602    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64007c09_5401_47c2_a0d4_853382b7a983.slice/crio-4eba969de04c2a3401eb11b53f7b6ef63c6436d3e8370ae1560f6ff8823417b3 WatchSource:0}: Error finding container 4eba969de04c2a3401eb11b53f7b6ef63c6436d3e8370ae1560f6ff8823417b3: Status 404 returned error can't find the container with id 4eba969de04c2a3401eb11b53f7b6ef63c6436d3e8370ae1560f6ff8823417b3
Sep 14 13:30:56.431887 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:56.431852    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t" event={"ID":"64007c09-5401-47c2-a0d4-853382b7a983","Type":"ContainerStarted","Data":"21f22f0537234f958315b4bf996ac70ed7ae30c0532e29f13cebeea8ce425203"}
Sep 14 13:30:56.432260 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:56.431914    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t"
Sep 14 13:30:56.432260 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:56.431935    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t" event={"ID":"64007c09-5401-47c2-a0d4-853382b7a983","Type":"ContainerStarted","Data":"4eba969de04c2a3401eb11b53f7b6ef63c6436d3e8370ae1560f6ff8823417b3"}
Sep 14 13:30:56.448618 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:30:56.448571    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t" podStartSLOduration=1.44855646 podStartE2EDuration="1.44855646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:30:56.447091664 +0000 UTC m=+1974.881767185" watchObservedRunningTime="2026-09-14 13:30:56.44855646 +0000 UTC m=+1974.883231979"
Sep 14 13:31:07.439468 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:07.439431    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t"
Sep 14 13:31:09.517056 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:09.517025    2557 generic.go:356] "Generic (PLEG): container finished" podID="bd99aea0-8666-4eb5-9f98-3f371211f521" containerID="86cbb1507615d12b5ecb165ae2fc217b37a38a85237a48bad013d488a934ee81" exitCode=0
Sep 14 13:31:09.517437 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:09.517100    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q" event={"ID":"bd99aea0-8666-4eb5-9f98-3f371211f521","Type":"ContainerDied","Data":"86cbb1507615d12b5ecb165ae2fc217b37a38a85237a48bad013d488a934ee81"}
Sep 14 13:31:09.517600 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:09.517583    2557 scope.go:117] "RemoveContainer" containerID="86cbb1507615d12b5ecb165ae2fc217b37a38a85237a48bad013d488a934ee81"
Sep 14 13:31:09.518679 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:09.518655    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:31:09.518778 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:09.518728    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:31:09.518831 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:09.518791    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 13:31:10.523896 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:10.523855    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q" event={"ID":"bd99aea0-8666-4eb5-9f98-3f371211f521","Type":"ContainerStarted","Data":"86b8267543d4930ec5205e920f869a2b6bdd8589baafa5a44910042dd8e1093a"}
Sep 14 13:31:11.360444 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:11.359859    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:31:11.360444 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:11.360135    2557 patch_prober.go:28] interesting pod/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Sep 14 13:31:11.360444 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:11.360181    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q" podUID="bd99aea0-8666-4eb5-9f98-3f371211f521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Sep 14 13:31:12.359705 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:12.359660    2557 patch_prober.go:28] interesting pod/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Sep 14 13:31:12.360214 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:12.359729    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q" podUID="bd99aea0-8666-4eb5-9f98-3f371211f521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Sep 14 13:31:13.359562 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:13.359521    2557 patch_prober.go:28] interesting pod/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Sep 14 13:31:13.359785 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:13.359593    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q" podUID="bd99aea0-8666-4eb5-9f98-3f371211f521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Sep 14 13:31:14.359773 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:14.359718    2557 patch_prober.go:28] interesting pod/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Sep 14 13:31:14.360268 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:14.359802    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q" podUID="bd99aea0-8666-4eb5-9f98-3f371211f521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Sep 14 13:31:15.359933 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:15.359893    2557 patch_prober.go:28] interesting pod/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Sep 14 13:31:15.360441 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:15.359959    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q" podUID="bd99aea0-8666-4eb5-9f98-3f371211f521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Sep 14 13:31:16.362638 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:16.362610    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:31:16.363091 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:16.362972    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:31:16.363643 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:16.363626    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:31:22.250172 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:22.250143    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zk2hb_692bcba1-e58a-4309-a7aa-4d13e276051b/payload-processing/0.log"
Sep 14 13:31:31.701558 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.701528    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"]
Sep 14 13:31:31.706239 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.706219    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:31.710455 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.710433    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c87a95\"/\"openshift-service-ca.crt\""
Sep 14 13:31:31.710455 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.710436    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c87a95\"/\"ipp-route-c87a95-kserve-self-signed-certs\""
Sep 14 13:31:31.710647 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.710490    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c87a95\"/\"default-dockercfg-qd5qj\""
Sep 14 13:31:31.710647 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.710493    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c87a95\"/\"kube-root-ca.crt\""
Sep 14 13:31:31.716591 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.716564    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"]
Sep 14 13:31:31.799474 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.799448    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-home\") pod \"ipp-route-c87a95-kserve-84b45974bc-5j9pl\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") " pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:31.799594 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.799487    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-dshm\") pod \"ipp-route-c87a95-kserve-84b45974bc-5j9pl\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") " pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:31.799594 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.799518    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmhgs\" (UniqueName: \"kubernetes.io/projected/86e40390-38aa-4015-a1dc-7434c8e08585-kube-api-access-lmhgs\") pod \"ipp-route-c87a95-kserve-84b45974bc-5j9pl\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") " pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:31.799594 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.799580    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e40390-38aa-4015-a1dc-7434c8e08585-tls-certs\") pod \"ipp-route-c87a95-kserve-84b45974bc-5j9pl\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") " pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:31.799703 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.799612    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-tmp-dir\") pod \"ipp-route-c87a95-kserve-84b45974bc-5j9pl\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") " pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:31.799703 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.799646    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-model-cache\") pod \"ipp-route-c87a95-kserve-84b45974bc-5j9pl\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") " pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:31.901095 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.901068    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmhgs\" (UniqueName: \"kubernetes.io/projected/86e40390-38aa-4015-a1dc-7434c8e08585-kube-api-access-lmhgs\") pod \"ipp-route-c87a95-kserve-84b45974bc-5j9pl\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") " pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:31.901330 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.901303    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e40390-38aa-4015-a1dc-7434c8e08585-tls-certs\") pod \"ipp-route-c87a95-kserve-84b45974bc-5j9pl\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") " pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:31.901487 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.901367    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-tmp-dir\") pod \"ipp-route-c87a95-kserve-84b45974bc-5j9pl\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") " pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:31.901487 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.901409    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-model-cache\") pod \"ipp-route-c87a95-kserve-84b45974bc-5j9pl\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") " pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:31.901487 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.901466    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-home\") pod \"ipp-route-c87a95-kserve-84b45974bc-5j9pl\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") " pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:31.901763 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.901500    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-dshm\") pod \"ipp-route-c87a95-kserve-84b45974bc-5j9pl\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") " pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:31.903218 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.902262    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-tmp-dir\") pod \"ipp-route-c87a95-kserve-84b45974bc-5j9pl\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") " pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:31.903218 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.902572    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-model-cache\") pod \"ipp-route-c87a95-kserve-84b45974bc-5j9pl\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") " pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:31.903715 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.903683    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-home\") pod \"ipp-route-c87a95-kserve-84b45974bc-5j9pl\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") " pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:31.905135 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.904326    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-dshm\") pod \"ipp-route-c87a95-kserve-84b45974bc-5j9pl\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") " pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:31.906236 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.905453    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e40390-38aa-4015-a1dc-7434c8e08585-tls-certs\") pod \"ipp-route-c87a95-kserve-84b45974bc-5j9pl\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") " pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:31.910027 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:31.910001    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmhgs\" (UniqueName: \"kubernetes.io/projected/86e40390-38aa-4015-a1dc-7434c8e08585-kube-api-access-lmhgs\") pod \"ipp-route-c87a95-kserve-84b45974bc-5j9pl\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") " pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:32.017236 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:32.017168    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:32.289142 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:32.288858    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"]
Sep 14 13:31:32.291808 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:31:32.291778    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86e40390_38aa_4015_a1dc_7434c8e08585.slice/crio-e0aa33525eaf05d28008db4245417e97707427e864a956dedb9f0c80e01b9caa WatchSource:0}: Error finding container e0aa33525eaf05d28008db4245417e97707427e864a956dedb9f0c80e01b9caa: Status 404 returned error can't find the container with id e0aa33525eaf05d28008db4245417e97707427e864a956dedb9f0c80e01b9caa
Sep 14 13:31:32.648285 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:32.648199    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl" event={"ID":"86e40390-38aa-4015-a1dc-7434c8e08585","Type":"ContainerStarted","Data":"abafedf7f13183d397d734520f5f1ddfe986c93983e601528487b85dc86675ec"}
Sep 14 13:31:32.648285 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:32.648247    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl" event={"ID":"86e40390-38aa-4015-a1dc-7434c8e08585","Type":"ContainerStarted","Data":"e0aa33525eaf05d28008db4245417e97707427e864a956dedb9f0c80e01b9caa"}
Sep 14 13:31:32.666707 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:32.666663    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl" podStartSLOduration=1.666648686 podStartE2EDuration="1.666648686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:31:32.665497287 +0000 UTC m=+2011.100172803" watchObservedRunningTime="2026-09-14 13:31:32.666648686 +0000 UTC m=+2011.101324204"
Sep 14 13:31:42.017918 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:42.017889    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:42.018393 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:42.017932    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:42.030787 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:42.030764    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:42.713239 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:42.713210    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:31:58.227343 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:31:58.227311    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zk2hb_692bcba1-e58a-4309-a7aa-4d13e276051b/payload-processing/0.log"
Sep 14 13:32:05.342455 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.342423    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg"]
Sep 14 13:32:05.345943 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.345921    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg"
Sep 14 13:32:05.349082 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.349058    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-8e2f31-dockercfg-fzhwr\""
Sep 14 13:32:05.349385 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.349148    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-8e2f31\""
Sep 14 13:32:05.357058 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.357037    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg"]
Sep 14 13:32:05.365643 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.365621    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4"]
Sep 14 13:32:05.369590 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.369571    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4"
Sep 14 13:32:05.377758 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.377712    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4"]
Sep 14 13:32:05.480444 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.480395    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4\" (UID: \"d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4"
Sep 14 13:32:05.480619 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.480542    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl7gj\" (UniqueName: \"kubernetes.io/projected/d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2-kube-api-access-wl7gj\") pod \"payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4\" (UID: \"d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4"
Sep 14 13:32:05.480679 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.480647    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shtgw\" (UniqueName: \"kubernetes.io/projected/e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0-kube-api-access-shtgw\") pod \"payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg\" (UID: \"e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg"
Sep 14 13:32:05.480722 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.480681    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg\" (UID: \"e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg"
Sep 14 13:32:05.581257 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.581225    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl7gj\" (UniqueName: \"kubernetes.io/projected/d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2-kube-api-access-wl7gj\") pod \"payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4\" (UID: \"d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4"
Sep 14 13:32:05.581422 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.581297    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shtgw\" (UniqueName: \"kubernetes.io/projected/e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0-kube-api-access-shtgw\") pod \"payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg\" (UID: \"e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg"
Sep 14 13:32:05.581422 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.581329    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg\" (UID: \"e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg"
Sep 14 13:32:05.581422 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.581349    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4\" (UID: \"d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4"
Sep 14 13:32:05.582083 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.582064    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4\" (UID: \"d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4"
Sep 14 13:32:05.582155 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.582117    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg\" (UID: \"e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg"
Sep 14 13:32:05.589559 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.589534    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl7gj\" (UniqueName: \"kubernetes.io/projected/d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2-kube-api-access-wl7gj\") pod \"payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4\" (UID: \"d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4"
Sep 14 13:32:05.589653 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.589601    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shtgw\" (UniqueName: \"kubernetes.io/projected/e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0-kube-api-access-shtgw\") pod \"payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg\" (UID: \"e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg"
Sep 14 13:32:05.657229 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.657169    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg"
Sep 14 13:32:05.681275 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.681247    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4"
Sep 14 13:32:05.935511 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.935484    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg"]
Sep 14 13:32:05.938322 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:32:05.938288    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b3c56e_ebcf_4e3e_aca5_c0a513fc70c0.slice/crio-92c9a433a23c29635add6e71b21d9a0b440dfc6b8550088b25008a20a908bf5d WatchSource:0}: Error finding container 92c9a433a23c29635add6e71b21d9a0b440dfc6b8550088b25008a20a908bf5d: Status 404 returned error can't find the container with id 92c9a433a23c29635add6e71b21d9a0b440dfc6b8550088b25008a20a908bf5d
Sep 14 13:32:05.975612 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:05.975481    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4"]
Sep 14 13:32:05.977858 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:32:05.977834    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e4a15a_0330_4db2_b0f3_c5ccbc79aca2.slice/crio-900b17b1c934635b5f4ce1f8d1636a6dd10bef84cb121ab020a3e42150edae94 WatchSource:0}: Error finding container 900b17b1c934635b5f4ce1f8d1636a6dd10bef84cb121ab020a3e42150edae94: Status 404 returned error can't find the container with id 900b17b1c934635b5f4ce1f8d1636a6dd10bef84cb121ab020a3e42150edae94
Sep 14 13:32:06.833350 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:06.833309    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4" event={"ID":"d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2","Type":"ContainerStarted","Data":"2f7d5b07365ae73760017f544c8c481b315966f678a8afc983998abfc6707274"}
Sep 14 13:32:06.833808 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:06.833408    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4" event={"ID":"d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2","Type":"ContainerStarted","Data":"900b17b1c934635b5f4ce1f8d1636a6dd10bef84cb121ab020a3e42150edae94"}
Sep 14 13:32:06.833808 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:06.833508    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4"
Sep 14 13:32:06.835033 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:06.835012    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg" event={"ID":"e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0","Type":"ContainerStarted","Data":"7dadb4d78656b44c7c0f6aad85f0f3d553c3037b01fa48521b9d48067c7bd652"}
Sep 14 13:32:06.835145 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:06.835059    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg" event={"ID":"e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0","Type":"ContainerStarted","Data":"92c9a433a23c29635add6e71b21d9a0b440dfc6b8550088b25008a20a908bf5d"}
Sep 14 13:32:06.835185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:06.835143    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg"
Sep 14 13:32:06.857667 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:06.857626    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4" podStartSLOduration=1.857612322 podStartE2EDuration="1.857612322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:32:06.856309465 +0000 UTC m=+2045.290984984" watchObservedRunningTime="2026-09-14 13:32:06.857612322 +0000 UTC m=+2045.292287841"
Sep 14 13:32:06.878609 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:06.878569    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg" podStartSLOduration=1.878555828 podStartE2EDuration="1.878555828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:32:06.877486631 +0000 UTC m=+2045.312162153" watchObservedRunningTime="2026-09-14 13:32:06.878555828 +0000 UTC m=+2045.313231347"
Sep 14 13:32:17.842213 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:17.842182    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4"
Sep 14 13:32:17.842725 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:17.842372    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg"
Sep 14 13:32:27.302781 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:27.302694    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e2f31-cjxvr"]
Sep 14 13:32:27.306385 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:27.306367    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e2f31-cjxvr"
Sep 14 13:32:27.309121 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:27.309102    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbzp\""
Sep 14 13:32:27.313732 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:27.313705    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e2f31-cjxvr"]
Sep 14 13:32:27.360679 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:27.360653    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08582c33-8d43-48b4-b81e-d7e2f74ef454-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8e2f31-cjxvr\" (UID: \"08582c33-8d43-48b4-b81e-d7e2f74ef454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e2f31-cjxvr"
Sep 14 13:32:27.461593 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:27.461546    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08582c33-8d43-48b4-b81e-d7e2f74ef454-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8e2f31-cjxvr\" (UID: \"08582c33-8d43-48b4-b81e-d7e2f74ef454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e2f31-cjxvr"
Sep 14 13:32:27.462313 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:27.462294    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08582c33-8d43-48b4-b81e-d7e2f74ef454-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8e2f31-cjxvr\" (UID: \"08582c33-8d43-48b4-b81e-d7e2f74ef454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e2f31-cjxvr"
Sep 14 13:32:27.620042 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:27.619966    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e2f31-cjxvr"
Sep 14 13:32:27.889842 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:27.889820    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e2f31-cjxvr"]
Sep 14 13:32:27.891819 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:32:27.891790    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08582c33_8d43_48b4_b81e_d7e2f74ef454.slice/crio-a87c82483e8c086726ab5abeeeb975a3deda1a45e2fe22b9862b42d592556fa3 WatchSource:0}: Error finding container a87c82483e8c086726ab5abeeeb975a3deda1a45e2fe22b9862b42d592556fa3: Status 404 returned error can't find the container with id a87c82483e8c086726ab5abeeeb975a3deda1a45e2fe22b9862b42d592556fa3
Sep 14 13:32:27.953410 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:27.953371    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e2f31-cjxvr" event={"ID":"08582c33-8d43-48b4-b81e-d7e2f74ef454","Type":"ContainerStarted","Data":"a87c82483e8c086726ab5abeeeb975a3deda1a45e2fe22b9862b42d592556fa3"}
Sep 14 13:32:28.961018 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:28.960982    2557 generic.go:356] "Generic (PLEG): container finished" podID="08582c33-8d43-48b4-b81e-d7e2f74ef454" containerID="9a9c590bf813876552a50abcc9d4b0ddcff973d7c71df2a91df4c5292258885e" exitCode=0
Sep 14 13:32:28.961388 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:28.961026    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e2f31-cjxvr" event={"ID":"08582c33-8d43-48b4-b81e-d7e2f74ef454","Type":"ContainerDied","Data":"9a9c590bf813876552a50abcc9d4b0ddcff973d7c71df2a91df4c5292258885e"}
Sep 14 13:32:30.105538 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:30.105515    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e2f31-cjxvr"
Sep 14 13:32:30.189575 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:30.189545    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08582c33-8d43-48b4-b81e-d7e2f74ef454-maas-api-service-ca\") pod \"08582c33-8d43-48b4-b81e-d7e2f74ef454\" (UID: \"08582c33-8d43-48b4-b81e-d7e2f74ef454\") "
Sep 14 13:32:30.189922 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:30.189900    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08582c33-8d43-48b4-b81e-d7e2f74ef454-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "08582c33-8d43-48b4-b81e-d7e2f74ef454" (UID: "08582c33-8d43-48b4-b81e-d7e2f74ef454"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:32:30.290339 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:30.290268    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08582c33-8d43-48b4-b81e-d7e2f74ef454-maas-api-service-ca\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:32:30.973587 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:30.973557    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e2f31-cjxvr"
Sep 14 13:32:30.973587 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:30.973578    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e2f31-cjxvr" event={"ID":"08582c33-8d43-48b4-b81e-d7e2f74ef454","Type":"ContainerDied","Data":"a87c82483e8c086726ab5abeeeb975a3deda1a45e2fe22b9862b42d592556fa3"}
Sep 14 13:32:30.973885 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:30.973650    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a87c82483e8c086726ab5abeeeb975a3deda1a45e2fe22b9862b42d592556fa3"
Sep 14 13:32:33.616300 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:33.616259    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4"]
Sep 14 13:32:33.616975 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:33.616485    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4" podUID="d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2" containerName="payload-processing" containerID="cri-o://2f7d5b07365ae73760017f544c8c481b315966f678a8afc983998abfc6707274" gracePeriod=30
Sep 14 13:32:33.628316 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:33.628288    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg"]
Sep 14 13:32:33.628562 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:33.628524    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg" podUID="e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0" containerName="payload-pre-processing" containerID="cri-o://7dadb4d78656b44c7c0f6aad85f0f3d553c3037b01fa48521b9d48067c7bd652" gracePeriod=30
Sep 14 13:32:33.798350 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:33.798270    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e2f31-cjxvr"]
Sep 14 13:32:33.801647 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:33.801625    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e2f31-cjxvr"]
Sep 14 13:32:33.928927 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:33.928906    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4"
Sep 14 13:32:33.931559 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:33.931540    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg"
Sep 14 13:32:33.992810 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:33.992779    2557 generic.go:356] "Generic (PLEG): container finished" podID="e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0" containerID="7dadb4d78656b44c7c0f6aad85f0f3d553c3037b01fa48521b9d48067c7bd652" exitCode=0
Sep 14 13:32:33.992989 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:33.992839    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg"
Sep 14 13:32:33.992989 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:33.992872    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg" event={"ID":"e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0","Type":"ContainerDied","Data":"7dadb4d78656b44c7c0f6aad85f0f3d553c3037b01fa48521b9d48067c7bd652"}
Sep 14 13:32:33.992989 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:33.992922    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg" event={"ID":"e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0","Type":"ContainerDied","Data":"92c9a433a23c29635add6e71b21d9a0b440dfc6b8550088b25008a20a908bf5d"}
Sep 14 13:32:33.992989 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:33.992945    2557 scope.go:117] "RemoveContainer" containerID="7dadb4d78656b44c7c0f6aad85f0f3d553c3037b01fa48521b9d48067c7bd652"
Sep 14 13:32:33.997611 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:33.997580    2557 generic.go:356] "Generic (PLEG): container finished" podID="d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2" containerID="2f7d5b07365ae73760017f544c8c481b315966f678a8afc983998abfc6707274" exitCode=0
Sep 14 13:32:33.997765 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:33.997631    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4"
Sep 14 13:32:33.997765 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:33.997661    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4" event={"ID":"d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2","Type":"ContainerDied","Data":"2f7d5b07365ae73760017f544c8c481b315966f678a8afc983998abfc6707274"}
Sep 14 13:32:33.997765 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:33.997713    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4" event={"ID":"d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2","Type":"ContainerDied","Data":"900b17b1c934635b5f4ce1f8d1636a6dd10bef84cb121ab020a3e42150edae94"}
Sep 14 13:32:34.005869 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:34.005855    2557 scope.go:117] "RemoveContainer" containerID="7dadb4d78656b44c7c0f6aad85f0f3d553c3037b01fa48521b9d48067c7bd652"
Sep 14 13:32:34.006117 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:32:34.006102    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dadb4d78656b44c7c0f6aad85f0f3d553c3037b01fa48521b9d48067c7bd652\": container with ID starting with 7dadb4d78656b44c7c0f6aad85f0f3d553c3037b01fa48521b9d48067c7bd652 not found: ID does not exist" containerID="7dadb4d78656b44c7c0f6aad85f0f3d553c3037b01fa48521b9d48067c7bd652"
Sep 14 13:32:34.006172 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:34.006126    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dadb4d78656b44c7c0f6aad85f0f3d553c3037b01fa48521b9d48067c7bd652"} err="failed to get container status \"7dadb4d78656b44c7c0f6aad85f0f3d553c3037b01fa48521b9d48067c7bd652\": rpc error: code = NotFound desc = could not find container \"7dadb4d78656b44c7c0f6aad85f0f3d553c3037b01fa48521b9d48067c7bd652\": container with ID starting with 7dadb4d78656b44c7c0f6aad85f0f3d553c3037b01fa48521b9d48067c7bd652 not found: ID does not exist"
Sep 14 13:32:34.006172 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:34.006141    2557 scope.go:117] "RemoveContainer" containerID="2f7d5b07365ae73760017f544c8c481b315966f678a8afc983998abfc6707274"
Sep 14 13:32:34.013840 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:34.013824    2557 scope.go:117] "RemoveContainer" containerID="2f7d5b07365ae73760017f544c8c481b315966f678a8afc983998abfc6707274"
Sep 14 13:32:34.014145 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:32:34.014128    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f7d5b07365ae73760017f544c8c481b315966f678a8afc983998abfc6707274\": container with ID starting with 2f7d5b07365ae73760017f544c8c481b315966f678a8afc983998abfc6707274 not found: ID does not exist" containerID="2f7d5b07365ae73760017f544c8c481b315966f678a8afc983998abfc6707274"
Sep 14 13:32:34.014202 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:34.014151    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f7d5b07365ae73760017f544c8c481b315966f678a8afc983998abfc6707274"} err="failed to get container status \"2f7d5b07365ae73760017f544c8c481b315966f678a8afc983998abfc6707274\": rpc error: code = NotFound desc = could not find container \"2f7d5b07365ae73760017f544c8c481b315966f678a8afc983998abfc6707274\": container with ID starting with 2f7d5b07365ae73760017f544c8c481b315966f678a8afc983998abfc6707274 not found: ID does not exist"
Sep 14 13:32:34.022439 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:34.022420    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shtgw\" (UniqueName: \"kubernetes.io/projected/e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0-kube-api-access-shtgw\") pod \"e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0\" (UID: \"e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0\") "
Sep 14 13:32:34.022520 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:34.022508    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl7gj\" (UniqueName: \"kubernetes.io/projected/d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2-kube-api-access-wl7gj\") pod \"d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2\" (UID: \"d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2\") "
Sep 14 13:32:34.022582 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:34.022527    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2-plugins-config-volume\") pod \"d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2\" (UID: \"d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2\") "
Sep 14 13:32:34.022582 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:34.022552    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0-plugins-config-volume\") pod \"e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0\" (UID: \"e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0\") "
Sep 14 13:32:34.022992 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:34.022966    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0" (UID: "e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:32:34.023088 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:34.022969    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2" (UID: "d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:32:34.024353 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:34.024332    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0-kube-api-access-shtgw" (OuterVolumeSpecName: "kube-api-access-shtgw") pod "e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0" (UID: "e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0"). InnerVolumeSpecName "kube-api-access-shtgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:32:34.024486 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:34.024470    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2-kube-api-access-wl7gj" (OuterVolumeSpecName: "kube-api-access-wl7gj") pod "d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2" (UID: "d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2"). InnerVolumeSpecName "kube-api-access-wl7gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:32:34.049456 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:34.049432    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08582c33-8d43-48b4-b81e-d7e2f74ef454" path="/var/lib/kubelet/pods/08582c33-8d43-48b4-b81e-d7e2f74ef454/volumes"
Sep 14 13:32:34.123868 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:34.123846    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shtgw\" (UniqueName: \"kubernetes.io/projected/e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0-kube-api-access-shtgw\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:32:34.123868 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:34.123868    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl7gj\" (UniqueName: \"kubernetes.io/projected/d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2-kube-api-access-wl7gj\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:32:34.123868 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:34.123878    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:32:34.124038 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:34.123886    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:32:34.312755 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:34.312710    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg"]
Sep 14 13:32:34.316936 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:34.316906    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e2f31-5bc4dc69c4-hftdg"]
Sep 14 13:32:34.328625 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:34.328604    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4"]
Sep 14 13:32:34.333942 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:34.333922    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8e2f31-5f66649d5c-n2sj4"]
Sep 14 13:32:36.049067 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:36.049033    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2" path="/var/lib/kubelet/pods/d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2/volumes"
Sep 14 13:32:36.049477 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:36.049361    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0" path="/var/lib/kubelet/pods/e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0/volumes"
Sep 14 13:32:43.706236 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:43.706199    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87a95-7wwds"]
Sep 14 13:32:43.706646 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:43.706631    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08582c33-8d43-48b4-b81e-d7e2f74ef454" containerName="revoke-keys"
Sep 14 13:32:43.706646 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:43.706646    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="08582c33-8d43-48b4-b81e-d7e2f74ef454" containerName="revoke-keys"
Sep 14 13:32:43.706723 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:43.706659    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0" containerName="payload-pre-processing"
Sep 14 13:32:43.706723 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:43.706664    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0" containerName="payload-pre-processing"
Sep 14 13:32:43.706723 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:43.706677    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2" containerName="payload-processing"
Sep 14 13:32:43.706723 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:43.706684    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2" containerName="payload-processing"
Sep 14 13:32:43.706877 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:43.706764    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0e4a15a-0330-4db2-b0f3-c5ccbc79aca2" containerName="payload-processing"
Sep 14 13:32:43.706877 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:43.706776    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="08582c33-8d43-48b4-b81e-d7e2f74ef454" containerName="revoke-keys"
Sep 14 13:32:43.706877 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:43.706786    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0b3c56e-ebcf-4e3e-aca5-c0a513fc70c0" containerName="payload-pre-processing"
Sep 14 13:32:43.709927 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:43.709910    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87a95-7wwds"
Sep 14 13:32:43.712663 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:43.712642    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbzp\""
Sep 14 13:32:43.718324 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:43.718231    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87a95-7wwds"]
Sep 14 13:32:43.805190 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:43.805162    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5bce4b4-d4a1-4222-bdf9-41a27d990432-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c87a95-7wwds\" (UID: \"d5bce4b4-d4a1-4222-bdf9-41a27d990432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87a95-7wwds"
Sep 14 13:32:43.906178 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:43.906147    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5bce4b4-d4a1-4222-bdf9-41a27d990432-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c87a95-7wwds\" (UID: \"d5bce4b4-d4a1-4222-bdf9-41a27d990432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87a95-7wwds"
Sep 14 13:32:43.906768 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:43.906724    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5bce4b4-d4a1-4222-bdf9-41a27d990432-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c87a95-7wwds\" (UID: \"d5bce4b4-d4a1-4222-bdf9-41a27d990432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87a95-7wwds"
Sep 14 13:32:44.022212 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:44.022151    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87a95-7wwds"
Sep 14 13:32:44.298535 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:44.298509    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87a95-7wwds"]
Sep 14 13:32:44.300367 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:32:44.300341    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5bce4b4_d4a1_4222_bdf9_41a27d990432.slice/crio-ebb33b52b9165d345befbe08c1c13afd90460ede9650a732966aa3af666765fc WatchSource:0}: Error finding container ebb33b52b9165d345befbe08c1c13afd90460ede9650a732966aa3af666765fc: Status 404 returned error can't find the container with id ebb33b52b9165d345befbe08c1c13afd90460ede9650a732966aa3af666765fc
Sep 14 13:32:45.065271 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:45.065239    2557 generic.go:356] "Generic (PLEG): container finished" podID="d5bce4b4-d4a1-4222-bdf9-41a27d990432" containerID="c656f35578b1abeb19f78448fc1878bb1b1bd31140a68949f019ed456fa0499e" exitCode=0
Sep 14 13:32:45.065588 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:45.065286    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87a95-7wwds" event={"ID":"d5bce4b4-d4a1-4222-bdf9-41a27d990432","Type":"ContainerDied","Data":"c656f35578b1abeb19f78448fc1878bb1b1bd31140a68949f019ed456fa0499e"}
Sep 14 13:32:45.065588 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:45.065326    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87a95-7wwds" event={"ID":"d5bce4b4-d4a1-4222-bdf9-41a27d990432","Type":"ContainerStarted","Data":"ebb33b52b9165d345befbe08c1c13afd90460ede9650a732966aa3af666765fc"}
Sep 14 13:32:46.206275 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:46.206249    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87a95-7wwds"
Sep 14 13:32:46.329983 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:46.329895    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5bce4b4-d4a1-4222-bdf9-41a27d990432-maas-api-service-ca\") pod \"d5bce4b4-d4a1-4222-bdf9-41a27d990432\" (UID: \"d5bce4b4-d4a1-4222-bdf9-41a27d990432\") "
Sep 14 13:32:46.330284 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:46.330261    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5bce4b4-d4a1-4222-bdf9-41a27d990432-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d5bce4b4-d4a1-4222-bdf9-41a27d990432" (UID: "d5bce4b4-d4a1-4222-bdf9-41a27d990432"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:32:46.431088 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:46.431062    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5bce4b4-d4a1-4222-bdf9-41a27d990432-maas-api-service-ca\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:32:47.077285 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:47.077253    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87a95-7wwds"
Sep 14 13:32:47.077434 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:47.077303    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87a95-7wwds" event={"ID":"d5bce4b4-d4a1-4222-bdf9-41a27d990432","Type":"ContainerDied","Data":"ebb33b52b9165d345befbe08c1c13afd90460ede9650a732966aa3af666765fc"}
Sep 14 13:32:47.077434 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:47.077350    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebb33b52b9165d345befbe08c1c13afd90460ede9650a732966aa3af666765fc"
Sep 14 13:32:59.232666 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:59.232630    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584"]
Sep 14 13:32:59.233244 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:59.232921    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584" podUID="74290e42-38b3-4436-a904-3c2960330824" containerName="payload-pre-processing" containerID="cri-o://27bca0bf8e49f41136dd2ab3faac4abdedeec1a28ac5b7d2b8081f0db7cff12a" gracePeriod=30
Sep 14 13:32:59.378576 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:59.378502    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87a95-7wwds"]
Sep 14 13:32:59.381869 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:59.381837    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87a95-7wwds"]
Sep 14 13:32:59.516800 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:59.516775    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584"
Sep 14 13:32:59.641864 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:59.641826    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2v28r\" (UniqueName: \"kubernetes.io/projected/74290e42-38b3-4436-a904-3c2960330824-kube-api-access-2v28r\") pod \"74290e42-38b3-4436-a904-3c2960330824\" (UID: \"74290e42-38b3-4436-a904-3c2960330824\") "
Sep 14 13:32:59.642024 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:59.641914    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74290e42-38b3-4436-a904-3c2960330824-plugins-config-volume\") pod \"74290e42-38b3-4436-a904-3c2960330824\" (UID: \"74290e42-38b3-4436-a904-3c2960330824\") "
Sep 14 13:32:59.642277 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:59.642251    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74290e42-38b3-4436-a904-3c2960330824-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "74290e42-38b3-4436-a904-3c2960330824" (UID: "74290e42-38b3-4436-a904-3c2960330824"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:32:59.644011 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:59.643991    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74290e42-38b3-4436-a904-3c2960330824-kube-api-access-2v28r" (OuterVolumeSpecName: "kube-api-access-2v28r") pod "74290e42-38b3-4436-a904-3c2960330824" (UID: "74290e42-38b3-4436-a904-3c2960330824"). InnerVolumeSpecName "kube-api-access-2v28r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:32:59.743494 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:59.743431    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74290e42-38b3-4436-a904-3c2960330824-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:32:59.743494 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:32:59.743454    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v28r\" (UniqueName: \"kubernetes.io/projected/74290e42-38b3-4436-a904-3c2960330824-kube-api-access-2v28r\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:33:00.050078 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:00.050004    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5bce4b4-d4a1-4222-bdf9-41a27d990432" path="/var/lib/kubelet/pods/d5bce4b4-d4a1-4222-bdf9-41a27d990432/volumes"
Sep 14 13:33:00.147468 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:00.147427    2557 generic.go:356] "Generic (PLEG): container finished" podID="74290e42-38b3-4436-a904-3c2960330824" containerID="27bca0bf8e49f41136dd2ab3faac4abdedeec1a28ac5b7d2b8081f0db7cff12a" exitCode=0
Sep 14 13:33:00.147610 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:00.147484    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584"
Sep 14 13:33:00.147610 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:00.147481    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584" event={"ID":"74290e42-38b3-4436-a904-3c2960330824","Type":"ContainerDied","Data":"27bca0bf8e49f41136dd2ab3faac4abdedeec1a28ac5b7d2b8081f0db7cff12a"}
Sep 14 13:33:00.147733 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:00.147607    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584" event={"ID":"74290e42-38b3-4436-a904-3c2960330824","Type":"ContainerDied","Data":"a9617940c8b3abd86f570b5b7c411c171201982bdc7a7eabadb0788d5aa16a3f"}
Sep 14 13:33:00.147733 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:00.147642    2557 scope.go:117] "RemoveContainer" containerID="27bca0bf8e49f41136dd2ab3faac4abdedeec1a28ac5b7d2b8081f0db7cff12a"
Sep 14 13:33:00.157407 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:00.157390    2557 scope.go:117] "RemoveContainer" containerID="27bca0bf8e49f41136dd2ab3faac4abdedeec1a28ac5b7d2b8081f0db7cff12a"
Sep 14 13:33:00.157699 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:33:00.157672    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27bca0bf8e49f41136dd2ab3faac4abdedeec1a28ac5b7d2b8081f0db7cff12a\": container with ID starting with 27bca0bf8e49f41136dd2ab3faac4abdedeec1a28ac5b7d2b8081f0db7cff12a not found: ID does not exist" containerID="27bca0bf8e49f41136dd2ab3faac4abdedeec1a28ac5b7d2b8081f0db7cff12a"
Sep 14 13:33:00.157771 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:00.157708    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27bca0bf8e49f41136dd2ab3faac4abdedeec1a28ac5b7d2b8081f0db7cff12a"} err="failed to get container status \"27bca0bf8e49f41136dd2ab3faac4abdedeec1a28ac5b7d2b8081f0db7cff12a\": rpc error: code = NotFound desc = could not find container \"27bca0bf8e49f41136dd2ab3faac4abdedeec1a28ac5b7d2b8081f0db7cff12a\": container with ID starting with 27bca0bf8e49f41136dd2ab3faac4abdedeec1a28ac5b7d2b8081f0db7cff12a not found: ID does not exist"
Sep 14 13:33:00.165961 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:00.165938    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584"]
Sep 14 13:33:00.169654 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:00.169633    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c87a95-6f9bdb7b78-q6584"]
Sep 14 13:33:02.049302 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:02.049271    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74290e42-38b3-4436-a904-3c2960330824" path="/var/lib/kubelet/pods/74290e42-38b3-4436-a904-3c2960330824/volumes"
Sep 14 13:33:02.216310 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:02.216276    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwqnf_62d953d6-3b86-4c95-9a47-9579d88993a7/ovn-acl-logging/0.log"
Sep 14 13:33:02.217225 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:02.217204    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwqnf_62d953d6-3b86-4c95-9a47-9579d88993a7/ovn-acl-logging/0.log"
Sep 14 13:33:02.630678 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:02.630654    2557 scope.go:117] "RemoveContainer" containerID="cd2ab91df0c7df70271748536c685de485ee071e1025729d9bf231633fd67156"
Sep 14 13:33:02.638621 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:02.638603    2557 scope.go:117] "RemoveContainer" containerID="425d26617440772fe96991df084778cb6f11d5de5a833bac7bad5b3afad5213d"
Sep 14 13:33:04.724101 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:04.724067    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"]
Sep 14 13:33:19.743289 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:19.743244    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl" podUID="86e40390-38aa-4015-a1dc-7434c8e08585" containerName="main" containerID="cri-o://abafedf7f13183d397d734520f5f1ddfe986c93983e601528487b85dc86675ec" gracePeriod=45
Sep 14 13:33:19.989579 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:19.989559    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:33:20.105198 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.105126    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-home\") pod \"86e40390-38aa-4015-a1dc-7434c8e08585\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") "
Sep 14 13:33:20.105198 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.105177    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-dshm\") pod \"86e40390-38aa-4015-a1dc-7434c8e08585\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") "
Sep 14 13:33:20.105394 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.105227    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e40390-38aa-4015-a1dc-7434c8e08585-tls-certs\") pod \"86e40390-38aa-4015-a1dc-7434c8e08585\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") "
Sep 14 13:33:20.105394 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.105269    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmhgs\" (UniqueName: \"kubernetes.io/projected/86e40390-38aa-4015-a1dc-7434c8e08585-kube-api-access-lmhgs\") pod \"86e40390-38aa-4015-a1dc-7434c8e08585\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") "
Sep 14 13:33:20.105394 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.105308    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-model-cache\") pod \"86e40390-38aa-4015-a1dc-7434c8e08585\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") "
Sep 14 13:33:20.105394 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.105338    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-tmp-dir\") pod \"86e40390-38aa-4015-a1dc-7434c8e08585\" (UID: \"86e40390-38aa-4015-a1dc-7434c8e08585\") "
Sep 14 13:33:20.105615 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.105417    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-home" (OuterVolumeSpecName: "home") pod "86e40390-38aa-4015-a1dc-7434c8e08585" (UID: "86e40390-38aa-4015-a1dc-7434c8e08585"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:33:20.105675 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.105608    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-model-cache" (OuterVolumeSpecName: "model-cache") pod "86e40390-38aa-4015-a1dc-7434c8e08585" (UID: "86e40390-38aa-4015-a1dc-7434c8e08585"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:33:20.105727 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.105660    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86e40390-38aa-4015-a1dc-7434c8e08585" (UID: "86e40390-38aa-4015-a1dc-7434c8e08585"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:33:20.105727 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.105681    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-model-cache\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:33:20.105727 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.105701    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-home\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:33:20.107416 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.107391    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-dshm" (OuterVolumeSpecName: "dshm") pod "86e40390-38aa-4015-a1dc-7434c8e08585" (UID: "86e40390-38aa-4015-a1dc-7434c8e08585"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:33:20.107543 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.107463    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86e40390-38aa-4015-a1dc-7434c8e08585-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86e40390-38aa-4015-a1dc-7434c8e08585" (UID: "86e40390-38aa-4015-a1dc-7434c8e08585"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:33:20.107543 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.107506    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e40390-38aa-4015-a1dc-7434c8e08585-kube-api-access-lmhgs" (OuterVolumeSpecName: "kube-api-access-lmhgs") pod "86e40390-38aa-4015-a1dc-7434c8e08585" (UID: "86e40390-38aa-4015-a1dc-7434c8e08585"). InnerVolumeSpecName "kube-api-access-lmhgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:33:20.206262 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.206239    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-dshm\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:33:20.206262 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.206259    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e40390-38aa-4015-a1dc-7434c8e08585-tls-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:33:20.206392 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.206270    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmhgs\" (UniqueName: \"kubernetes.io/projected/86e40390-38aa-4015-a1dc-7434c8e08585-kube-api-access-lmhgs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:33:20.206392 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.206296    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e40390-38aa-4015-a1dc-7434c8e08585-tmp-dir\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:33:20.255270 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.255244    2557 generic.go:356] "Generic (PLEG): container finished" podID="86e40390-38aa-4015-a1dc-7434c8e08585" containerID="abafedf7f13183d397d734520f5f1ddfe986c93983e601528487b85dc86675ec" exitCode=0
Sep 14 13:33:20.255362 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.255294    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"
Sep 14 13:33:20.255362 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.255327    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl" event={"ID":"86e40390-38aa-4015-a1dc-7434c8e08585","Type":"ContainerDied","Data":"abafedf7f13183d397d734520f5f1ddfe986c93983e601528487b85dc86675ec"}
Sep 14 13:33:20.255437 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.255374    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl" event={"ID":"86e40390-38aa-4015-a1dc-7434c8e08585","Type":"ContainerDied","Data":"e0aa33525eaf05d28008db4245417e97707427e864a956dedb9f0c80e01b9caa"}
Sep 14 13:33:20.255437 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.255394    2557 scope.go:117] "RemoveContainer" containerID="abafedf7f13183d397d734520f5f1ddfe986c93983e601528487b85dc86675ec"
Sep 14 13:33:20.264183 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.264167    2557 scope.go:117] "RemoveContainer" containerID="abafedf7f13183d397d734520f5f1ddfe986c93983e601528487b85dc86675ec"
Sep 14 13:33:20.264465 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:33:20.264441    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abafedf7f13183d397d734520f5f1ddfe986c93983e601528487b85dc86675ec\": container with ID starting with abafedf7f13183d397d734520f5f1ddfe986c93983e601528487b85dc86675ec not found: ID does not exist" containerID="abafedf7f13183d397d734520f5f1ddfe986c93983e601528487b85dc86675ec"
Sep 14 13:33:20.264508 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.264475    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abafedf7f13183d397d734520f5f1ddfe986c93983e601528487b85dc86675ec"} err="failed to get container status \"abafedf7f13183d397d734520f5f1ddfe986c93983e601528487b85dc86675ec\": rpc error: code = NotFound desc = could not find container \"abafedf7f13183d397d734520f5f1ddfe986c93983e601528487b85dc86675ec\": container with ID starting with abafedf7f13183d397d734520f5f1ddfe986c93983e601528487b85dc86675ec not found: ID does not exist"
Sep 14 13:33:20.275790 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.275769    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"]
Sep 14 13:33:20.279727 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:20.279709    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-c87a95/ipp-route-c87a95-kserve-84b45974bc-5j9pl"]
Sep 14 13:33:22.048794 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:22.048761    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86e40390-38aa-4015-a1dc-7434c8e08585" path="/var/lib/kubelet/pods/86e40390-38aa-4015-a1dc-7434c8e08585/volumes"
Sep 14 13:33:48.174824 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:48.174792    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"]
Sep 14 13:33:48.175373 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:48.174999    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k" podUID="17ab6ae2-e510-4102-8efc-40cb5c386af3" containerName="istio-proxy" containerID="cri-o://d41b8d039ae4ddc73d1c6f2dc48fb585a585845cfb935d7146cf0c72d873c7e2" gracePeriod=30
Sep 14 13:33:48.539531 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:48.539489    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-27ae76-mmh98"]
Sep 14 13:33:48.539920 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:48.539902    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5bce4b4-d4a1-4222-bdf9-41a27d990432" containerName="revoke-keys"
Sep 14 13:33:48.539920 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:48.539918    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5bce4b4-d4a1-4222-bdf9-41a27d990432" containerName="revoke-keys"
Sep 14 13:33:48.540110 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:48.539933    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86e40390-38aa-4015-a1dc-7434c8e08585" containerName="main"
Sep 14 13:33:48.540110 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:48.539938    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="86e40390-38aa-4015-a1dc-7434c8e08585" containerName="main"
Sep 14 13:33:48.540110 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:48.539945    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74290e42-38b3-4436-a904-3c2960330824" containerName="payload-pre-processing"
Sep 14 13:33:48.540110 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:48.539950    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="74290e42-38b3-4436-a904-3c2960330824" containerName="payload-pre-processing"
Sep 14 13:33:48.540110 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:48.540016    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="86e40390-38aa-4015-a1dc-7434c8e08585" containerName="main"
Sep 14 13:33:48.540110 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:48.540022    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5bce4b4-d4a1-4222-bdf9-41a27d990432" containerName="revoke-keys"
Sep 14 13:33:48.540110 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:48.540030    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="74290e42-38b3-4436-a904-3c2960330824" containerName="payload-pre-processing"
Sep 14 13:33:48.543123 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:48.543102    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-27ae76-mmh98"
Sep 14 13:33:48.546069 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:48.546051    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbzp\""
Sep 14 13:33:48.551841 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:48.551819    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-27ae76-mmh98"]
Sep 14 13:33:48.644392 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:48.644352    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32056bb8-e942-4ce0-80ad-e8f4bc9d80b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-27ae76-mmh98\" (UID: \"32056bb8-e942-4ce0-80ad-e8f4bc9d80b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-27ae76-mmh98"
Sep 14 13:33:48.745550 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:48.745513    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32056bb8-e942-4ce0-80ad-e8f4bc9d80b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-27ae76-mmh98\" (UID: \"32056bb8-e942-4ce0-80ad-e8f4bc9d80b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-27ae76-mmh98"
Sep 14 13:33:48.746161 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:48.746139    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32056bb8-e942-4ce0-80ad-e8f4bc9d80b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-27ae76-mmh98\" (UID: \"32056bb8-e942-4ce0-80ad-e8f4bc9d80b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-27ae76-mmh98"
Sep 14 13:33:48.854283 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:48.854207    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-27ae76-mmh98"
Sep 14 13:33:49.126018 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:49.125994    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-27ae76-mmh98"]
Sep 14 13:33:49.127762 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:33:49.127726    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32056bb8_e942_4ce0_80ad_e8f4bc9d80b5.slice/crio-1c88eeda8cab15b993e5b8d24b1f398cc053e7f0470da60512f11fc32edd0b2d WatchSource:0}: Error finding container 1c88eeda8cab15b993e5b8d24b1f398cc053e7f0470da60512f11fc32edd0b2d: Status 404 returned error can't find the container with id 1c88eeda8cab15b993e5b8d24b1f398cc053e7f0470da60512f11fc32edd0b2d
Sep 14 13:33:49.415423 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:49.415343    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-27ae76-mmh98" event={"ID":"32056bb8-e942-4ce0-80ad-e8f4bc9d80b5","Type":"ContainerStarted","Data":"1c88eeda8cab15b993e5b8d24b1f398cc053e7f0470da60512f11fc32edd0b2d"}
Sep 14 13:33:50.422589 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:50.422552    2557 generic.go:356] "Generic (PLEG): container finished" podID="32056bb8-e942-4ce0-80ad-e8f4bc9d80b5" containerID="25a096f514ca29a412d72e431293accf6b683f23c2bc3be7c5c63f28636d31d9" exitCode=0
Sep 14 13:33:50.423034 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:50.422633    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-27ae76-mmh98" event={"ID":"32056bb8-e942-4ce0-80ad-e8f4bc9d80b5","Type":"ContainerDied","Data":"25a096f514ca29a412d72e431293accf6b683f23c2bc3be7c5c63f28636d31d9"}
Sep 14 13:33:51.563209 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:51.563185    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-27ae76-mmh98"
Sep 14 13:33:51.671104 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:51.671077    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32056bb8-e942-4ce0-80ad-e8f4bc9d80b5-maas-api-service-ca\") pod \"32056bb8-e942-4ce0-80ad-e8f4bc9d80b5\" (UID: \"32056bb8-e942-4ce0-80ad-e8f4bc9d80b5\") "
Sep 14 13:33:51.671438 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:51.671418    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32056bb8-e942-4ce0-80ad-e8f4bc9d80b5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "32056bb8-e942-4ce0-80ad-e8f4bc9d80b5" (UID: "32056bb8-e942-4ce0-80ad-e8f4bc9d80b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:33:51.771942 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:51.771907    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32056bb8-e942-4ce0-80ad-e8f4bc9d80b5-maas-api-service-ca\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:33:52.434645 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:52.434572    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-27ae76-mmh98"
Sep 14 13:33:52.434645 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:52.434584    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-27ae76-mmh98" event={"ID":"32056bb8-e942-4ce0-80ad-e8f4bc9d80b5","Type":"ContainerDied","Data":"1c88eeda8cab15b993e5b8d24b1f398cc053e7f0470da60512f11fc32edd0b2d"}
Sep 14 13:33:52.434645 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:52.434626    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c88eeda8cab15b993e5b8d24b1f398cc053e7f0470da60512f11fc32edd0b2d"
Sep 14 13:33:53.433718 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.433659    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:33:53.441520 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.441489    2557 generic.go:356] "Generic (PLEG): container finished" podID="17ab6ae2-e510-4102-8efc-40cb5c386af3" containerID="d41b8d039ae4ddc73d1c6f2dc48fb585a585845cfb935d7146cf0c72d873c7e2" exitCode=0
Sep 14 13:33:53.441642 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.441525    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"
Sep 14 13:33:53.441642 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.441575    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k" event={"ID":"17ab6ae2-e510-4102-8efc-40cb5c386af3","Type":"ContainerDied","Data":"d41b8d039ae4ddc73d1c6f2dc48fb585a585845cfb935d7146cf0c72d873c7e2"}
Sep 14 13:33:53.441642 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.441635    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k" event={"ID":"17ab6ae2-e510-4102-8efc-40cb5c386af3","Type":"ContainerDied","Data":"677f1bf0903986946d684108a75f4dddf7a3e8e7fe2d06b708568957f317d412"}
Sep 14 13:33:53.441837 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.441667    2557 scope.go:117] "RemoveContainer" containerID="d41b8d039ae4ddc73d1c6f2dc48fb585a585845cfb935d7146cf0c72d873c7e2"
Sep 14 13:33:53.453580 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.453561    2557 scope.go:117] "RemoveContainer" containerID="ae2cae829194cdc9bc0f7b7574392058a9ae9bae44dd560cf6f50798f148c6b4"
Sep 14 13:33:53.461767 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.461750    2557 scope.go:117] "RemoveContainer" containerID="d41b8d039ae4ddc73d1c6f2dc48fb585a585845cfb935d7146cf0c72d873c7e2"
Sep 14 13:33:53.462029 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:33:53.462011    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d41b8d039ae4ddc73d1c6f2dc48fb585a585845cfb935d7146cf0c72d873c7e2\": container with ID starting with d41b8d039ae4ddc73d1c6f2dc48fb585a585845cfb935d7146cf0c72d873c7e2 not found: ID does not exist" containerID="d41b8d039ae4ddc73d1c6f2dc48fb585a585845cfb935d7146cf0c72d873c7e2"
Sep 14 13:33:53.462101 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.462043    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d41b8d039ae4ddc73d1c6f2dc48fb585a585845cfb935d7146cf0c72d873c7e2"} err="failed to get container status \"d41b8d039ae4ddc73d1c6f2dc48fb585a585845cfb935d7146cf0c72d873c7e2\": rpc error: code = NotFound desc = could not find container \"d41b8d039ae4ddc73d1c6f2dc48fb585a585845cfb935d7146cf0c72d873c7e2\": container with ID starting with d41b8d039ae4ddc73d1c6f2dc48fb585a585845cfb935d7146cf0c72d873c7e2 not found: ID does not exist"
Sep 14 13:33:53.462101 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.462068    2557 scope.go:117] "RemoveContainer" containerID="ae2cae829194cdc9bc0f7b7574392058a9ae9bae44dd560cf6f50798f148c6b4"
Sep 14 13:33:53.462328 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:33:53.462311    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae2cae829194cdc9bc0f7b7574392058a9ae9bae44dd560cf6f50798f148c6b4\": container with ID starting with ae2cae829194cdc9bc0f7b7574392058a9ae9bae44dd560cf6f50798f148c6b4 not found: ID does not exist" containerID="ae2cae829194cdc9bc0f7b7574392058a9ae9bae44dd560cf6f50798f148c6b4"
Sep 14 13:33:53.462386 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.462336    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2cae829194cdc9bc0f7b7574392058a9ae9bae44dd560cf6f50798f148c6b4"} err="failed to get container status \"ae2cae829194cdc9bc0f7b7574392058a9ae9bae44dd560cf6f50798f148c6b4\": rpc error: code = NotFound desc = could not find container \"ae2cae829194cdc9bc0f7b7574392058a9ae9bae44dd560cf6f50798f148c6b4\": container with ID starting with ae2cae829194cdc9bc0f7b7574392058a9ae9bae44dd560cf6f50798f148c6b4 not found: ID does not exist"
Sep 14 13:33:53.483346 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.483325    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-workload-certs\") pod \"17ab6ae2-e510-4102-8efc-40cb5c386af3\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") "
Sep 14 13:33:53.483443 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.483357    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-data\") pod \"17ab6ae2-e510-4102-8efc-40cb5c386af3\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") "
Sep 14 13:33:53.483443 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.483376    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-podinfo\") pod \"17ab6ae2-e510-4102-8efc-40cb5c386af3\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") "
Sep 14 13:33:53.483443 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.483401    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-credential-socket\") pod \"17ab6ae2-e510-4102-8efc-40cb5c386af3\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") "
Sep 14 13:33:53.483443 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.483437    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-envoy\") pod \"17ab6ae2-e510-4102-8efc-40cb5c386af3\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") "
Sep 14 13:33:53.483647 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.483464    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-workload-socket\") pod \"17ab6ae2-e510-4102-8efc-40cb5c386af3\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") "
Sep 14 13:33:53.483647 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.483501    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9sfl\" (UniqueName: \"kubernetes.io/projected/17ab6ae2-e510-4102-8efc-40cb5c386af3-kube-api-access-q9sfl\") pod \"17ab6ae2-e510-4102-8efc-40cb5c386af3\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") "
Sep 14 13:33:53.483647 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.483577    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ab6ae2-e510-4102-8efc-40cb5c386af3-istiod-ca-cert\") pod \"17ab6ae2-e510-4102-8efc-40cb5c386af3\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") "
Sep 14 13:33:53.483647 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.483612    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-token\") pod \"17ab6ae2-e510-4102-8efc-40cb5c386af3\" (UID: \"17ab6ae2-e510-4102-8efc-40cb5c386af3\") "
Sep 14 13:33:53.483647 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.483614    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "17ab6ae2-e510-4102-8efc-40cb5c386af3" (UID: "17ab6ae2-e510-4102-8efc-40cb5c386af3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:33:53.483935 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.483649    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-data" (OuterVolumeSpecName: "istio-data") pod "17ab6ae2-e510-4102-8efc-40cb5c386af3" (UID: "17ab6ae2-e510-4102-8efc-40cb5c386af3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:33:53.483996 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.483962    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-workload-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:33:53.483996 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.483983    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-data\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:33:53.484102 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.484016    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ab6ae2-e510-4102-8efc-40cb5c386af3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "17ab6ae2-e510-4102-8efc-40cb5c386af3" (UID: "17ab6ae2-e510-4102-8efc-40cb5c386af3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:33:53.484102 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.484027    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "17ab6ae2-e510-4102-8efc-40cb5c386af3" (UID: "17ab6ae2-e510-4102-8efc-40cb5c386af3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:33:53.484255 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.484228    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "17ab6ae2-e510-4102-8efc-40cb5c386af3" (UID: "17ab6ae2-e510-4102-8efc-40cb5c386af3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:33:53.485579 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.485559    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "17ab6ae2-e510-4102-8efc-40cb5c386af3" (UID: "17ab6ae2-e510-4102-8efc-40cb5c386af3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 13:33:53.485791 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.485730    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-token" (OuterVolumeSpecName: "istio-token") pod "17ab6ae2-e510-4102-8efc-40cb5c386af3" (UID: "17ab6ae2-e510-4102-8efc-40cb5c386af3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:33:53.486037 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.486015    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "17ab6ae2-e510-4102-8efc-40cb5c386af3" (UID: "17ab6ae2-e510-4102-8efc-40cb5c386af3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:33:53.486254 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.486233    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ab6ae2-e510-4102-8efc-40cb5c386af3-kube-api-access-q9sfl" (OuterVolumeSpecName: "kube-api-access-q9sfl") pod "17ab6ae2-e510-4102-8efc-40cb5c386af3" (UID: "17ab6ae2-e510-4102-8efc-40cb5c386af3"). InnerVolumeSpecName "kube-api-access-q9sfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:33:53.585312 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.585290    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ab6ae2-e510-4102-8efc-40cb5c386af3-istiod-ca-cert\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:33:53.585312 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.585313    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-token\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:33:53.585431 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.585322    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-podinfo\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:33:53.585431 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.585329    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-credential-socket\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:33:53.585431 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.585338    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-istio-envoy\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:33:53.585431 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.585346    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ab6ae2-e510-4102-8efc-40cb5c386af3-workload-socket\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:33:53.585431 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.585353    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9sfl\" (UniqueName: \"kubernetes.io/projected/17ab6ae2-e510-4102-8efc-40cb5c386af3-kube-api-access-q9sfl\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:33:53.768604 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.768581    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"]
Sep 14 13:33:53.771340 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:53.771314    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c87a95-openshift-default-7656478d8d-btv4k"]
Sep 14 13:33:54.049042 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:54.048966    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17ab6ae2-e510-4102-8efc-40cb5c386af3" path="/var/lib/kubelet/pods/17ab6ae2-e510-4102-8efc-40cb5c386af3/volumes"
Sep 14 13:33:59.065182 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:59.065096    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t"]
Sep 14 13:33:59.065617 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:59.065325    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t" podUID="64007c09-5401-47c2-a0d4-853382b7a983" containerName="payload-pre-processing" containerID="cri-o://21f22f0537234f958315b4bf996ac70ed7ae30c0532e29f13cebeea8ce425203" gracePeriod=30
Sep 14 13:33:59.212904 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:59.212869    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-27ae76-mmh98"]
Sep 14 13:33:59.216371 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:59.216342    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-27ae76-mmh98"]
Sep 14 13:33:59.310903 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:59.310876    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t"
Sep 14 13:33:59.432385 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:59.432309    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78fpv\" (UniqueName: \"kubernetes.io/projected/64007c09-5401-47c2-a0d4-853382b7a983-kube-api-access-78fpv\") pod \"64007c09-5401-47c2-a0d4-853382b7a983\" (UID: \"64007c09-5401-47c2-a0d4-853382b7a983\") "
Sep 14 13:33:59.432536 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:59.432454    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64007c09-5401-47c2-a0d4-853382b7a983-plugins-config-volume\") pod \"64007c09-5401-47c2-a0d4-853382b7a983\" (UID: \"64007c09-5401-47c2-a0d4-853382b7a983\") "
Sep 14 13:33:59.432804 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:59.432781    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64007c09-5401-47c2-a0d4-853382b7a983-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "64007c09-5401-47c2-a0d4-853382b7a983" (UID: "64007c09-5401-47c2-a0d4-853382b7a983"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:33:59.434435 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:59.434411    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64007c09-5401-47c2-a0d4-853382b7a983-kube-api-access-78fpv" (OuterVolumeSpecName: "kube-api-access-78fpv") pod "64007c09-5401-47c2-a0d4-853382b7a983" (UID: "64007c09-5401-47c2-a0d4-853382b7a983"). InnerVolumeSpecName "kube-api-access-78fpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:33:59.480235 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:59.480196    2557 generic.go:356] "Generic (PLEG): container finished" podID="64007c09-5401-47c2-a0d4-853382b7a983" containerID="21f22f0537234f958315b4bf996ac70ed7ae30c0532e29f13cebeea8ce425203" exitCode=0
Sep 14 13:33:59.480396 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:59.480282    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t"
Sep 14 13:33:59.480396 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:59.480307    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t" event={"ID":"64007c09-5401-47c2-a0d4-853382b7a983","Type":"ContainerDied","Data":"21f22f0537234f958315b4bf996ac70ed7ae30c0532e29f13cebeea8ce425203"}
Sep 14 13:33:59.480396 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:59.480393    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t" event={"ID":"64007c09-5401-47c2-a0d4-853382b7a983","Type":"ContainerDied","Data":"4eba969de04c2a3401eb11b53f7b6ef63c6436d3e8370ae1560f6ff8823417b3"}
Sep 14 13:33:59.480577 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:59.480439    2557 scope.go:117] "RemoveContainer" containerID="21f22f0537234f958315b4bf996ac70ed7ae30c0532e29f13cebeea8ce425203"
Sep 14 13:33:59.492445 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:59.492424    2557 scope.go:117] "RemoveContainer" containerID="21f22f0537234f958315b4bf996ac70ed7ae30c0532e29f13cebeea8ce425203"
Sep 14 13:33:59.492834 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:33:59.492807    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21f22f0537234f958315b4bf996ac70ed7ae30c0532e29f13cebeea8ce425203\": container with ID starting with 21f22f0537234f958315b4bf996ac70ed7ae30c0532e29f13cebeea8ce425203 not found: ID does not exist" containerID="21f22f0537234f958315b4bf996ac70ed7ae30c0532e29f13cebeea8ce425203"
Sep 14 13:33:59.492945 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:59.492842    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21f22f0537234f958315b4bf996ac70ed7ae30c0532e29f13cebeea8ce425203"} err="failed to get container status \"21f22f0537234f958315b4bf996ac70ed7ae30c0532e29f13cebeea8ce425203\": rpc error: code = NotFound desc = could not find container \"21f22f0537234f958315b4bf996ac70ed7ae30c0532e29f13cebeea8ce425203\": container with ID starting with 21f22f0537234f958315b4bf996ac70ed7ae30c0532e29f13cebeea8ce425203 not found: ID does not exist"
Sep 14 13:33:59.504825 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:59.504805    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t"]
Sep 14 13:33:59.509305 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:59.509286    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-27ae76-65d4b46858-7b66t"]
Sep 14 13:33:59.533378 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:59.533358    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78fpv\" (UniqueName: \"kubernetes.io/projected/64007c09-5401-47c2-a0d4-853382b7a983-kube-api-access-78fpv\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:33:59.533378 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:33:59.533376    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64007c09-5401-47c2-a0d4-853382b7a983-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:00.048330 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:00.048301    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32056bb8-e942-4ce0-80ad-e8f4bc9d80b5" path="/var/lib/kubelet/pods/32056bb8-e942-4ce0-80ad-e8f4bc9d80b5/volumes"
Sep 14 13:34:00.048669 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:00.048657    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64007c09-5401-47c2-a0d4-853382b7a983" path="/var/lib/kubelet/pods/64007c09-5401-47c2-a0d4-853382b7a983/volumes"
Sep 14 13:34:05.980690 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:05.980655    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"]
Sep 14 13:34:05.981096 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:05.980973    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q" podUID="bd99aea0-8666-4eb5-9f98-3f371211f521" containerName="istio-proxy" containerID="cri-o://86b8267543d4930ec5205e920f869a2b6bdd8589baafa5a44910042dd8e1093a" gracePeriod=30
Sep 14 13:34:06.333629 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:06.333544    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14f3af-m5whd"]
Sep 14 13:34:06.333929 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:06.333915    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32056bb8-e942-4ce0-80ad-e8f4bc9d80b5" containerName="revoke-keys"
Sep 14 13:34:06.333983 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:06.333929    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="32056bb8-e942-4ce0-80ad-e8f4bc9d80b5" containerName="revoke-keys"
Sep 14 13:34:06.333983 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:06.333940    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17ab6ae2-e510-4102-8efc-40cb5c386af3" containerName="istio-proxy"
Sep 14 13:34:06.333983 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:06.333945    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="17ab6ae2-e510-4102-8efc-40cb5c386af3" containerName="istio-proxy"
Sep 14 13:34:06.333983 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:06.333953    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17ab6ae2-e510-4102-8efc-40cb5c386af3" containerName="istio-proxy"
Sep 14 13:34:06.333983 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:06.333958    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="17ab6ae2-e510-4102-8efc-40cb5c386af3" containerName="istio-proxy"
Sep 14 13:34:06.333983 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:06.333966    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64007c09-5401-47c2-a0d4-853382b7a983" containerName="payload-pre-processing"
Sep 14 13:34:06.333983 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:06.333971    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="64007c09-5401-47c2-a0d4-853382b7a983" containerName="payload-pre-processing"
Sep 14 13:34:06.334242 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:06.334050    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="17ab6ae2-e510-4102-8efc-40cb5c386af3" containerName="istio-proxy"
Sep 14 13:34:06.334242 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:06.334056    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="64007c09-5401-47c2-a0d4-853382b7a983" containerName="payload-pre-processing"
Sep 14 13:34:06.334242 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:06.334062    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="32056bb8-e942-4ce0-80ad-e8f4bc9d80b5" containerName="revoke-keys"
Sep 14 13:34:06.334242 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:06.334069    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="17ab6ae2-e510-4102-8efc-40cb5c386af3" containerName="istio-proxy"
Sep 14 13:34:06.338621 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:06.338604    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14f3af-m5whd"
Sep 14 13:34:06.341379 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:06.341355    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbzp\""
Sep 14 13:34:06.344186 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:06.344155    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14f3af-m5whd"]
Sep 14 13:34:06.485059 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:06.485028    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8e99b99-a1e9-4713-b932-fa099f41b360-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-14f3af-m5whd\" (UID: \"a8e99b99-a1e9-4713-b932-fa099f41b360\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14f3af-m5whd"
Sep 14 13:34:06.586530 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:06.586448    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8e99b99-a1e9-4713-b932-fa099f41b360-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-14f3af-m5whd\" (UID: \"a8e99b99-a1e9-4713-b932-fa099f41b360\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14f3af-m5whd"
Sep 14 13:34:06.587103 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:06.587084    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8e99b99-a1e9-4713-b932-fa099f41b360-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-14f3af-m5whd\" (UID: \"a8e99b99-a1e9-4713-b932-fa099f41b360\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14f3af-m5whd"
Sep 14 13:34:06.651129 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:06.651107    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14f3af-m5whd"
Sep 14 13:34:06.923963 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:06.923937    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14f3af-m5whd"]
Sep 14 13:34:06.925292 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:34:06.925268    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e99b99_a1e9_4713_b932_fa099f41b360.slice/crio-b1c0521bb09f5a76d4c89930861a6cc5b1347e1d9ad4ac20674040f0e87e015e WatchSource:0}: Error finding container b1c0521bb09f5a76d4c89930861a6cc5b1347e1d9ad4ac20674040f0e87e015e: Status 404 returned error can't find the container with id b1c0521bb09f5a76d4c89930861a6cc5b1347e1d9ad4ac20674040f0e87e015e
Sep 14 13:34:07.524316 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:07.524285    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14f3af-m5whd" event={"ID":"a8e99b99-a1e9-4713-b932-fa099f41b360","Type":"ContainerStarted","Data":"9a54700428603e7690988c64c7a892e9f5100b2389410e5fe5d79614e944d3be"}
Sep 14 13:34:07.524682 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:07.524334    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14f3af-m5whd" event={"ID":"a8e99b99-a1e9-4713-b932-fa099f41b360","Type":"ContainerStarted","Data":"b1c0521bb09f5a76d4c89930861a6cc5b1347e1d9ad4ac20674040f0e87e015e"}
Sep 14 13:34:07.541104 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:07.541063    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14f3af-m5whd" podStartSLOduration=1.541050032 podStartE2EDuration="1.541050032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:34:07.539223912 +0000 UTC m=+2165.973899430" watchObservedRunningTime="2026-09-14 13:34:07.541050032 +0000 UTC m=+2165.975725550"
Sep 14 13:34:08.531185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:08.531152    2557 generic.go:356] "Generic (PLEG): container finished" podID="a8e99b99-a1e9-4713-b932-fa099f41b360" containerID="9a54700428603e7690988c64c7a892e9f5100b2389410e5fe5d79614e944d3be" exitCode=0
Sep 14 13:34:08.531532 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:08.531192    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14f3af-m5whd" event={"ID":"a8e99b99-a1e9-4713-b932-fa099f41b360","Type":"ContainerDied","Data":"9a54700428603e7690988c64c7a892e9f5100b2389410e5fe5d79614e944d3be"}
Sep 14 13:34:09.668381 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:09.668357    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14f3af-m5whd"
Sep 14 13:34:09.810901 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:09.810829    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8e99b99-a1e9-4713-b932-fa099f41b360-maas-api-service-ca\") pod \"a8e99b99-a1e9-4713-b932-fa099f41b360\" (UID: \"a8e99b99-a1e9-4713-b932-fa099f41b360\") "
Sep 14 13:34:09.811212 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:09.811193    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e99b99-a1e9-4713-b932-fa099f41b360-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a8e99b99-a1e9-4713-b932-fa099f41b360" (UID: "a8e99b99-a1e9-4713-b932-fa099f41b360"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:34:09.912340 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:09.912316    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8e99b99-a1e9-4713-b932-fa099f41b360-maas-api-service-ca\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:10.542280 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:10.542254    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14f3af-m5whd"
Sep 14 13:34:10.542445 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:10.542249    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14f3af-m5whd" event={"ID":"a8e99b99-a1e9-4713-b932-fa099f41b360","Type":"ContainerDied","Data":"b1c0521bb09f5a76d4c89930861a6cc5b1347e1d9ad4ac20674040f0e87e015e"}
Sep 14 13:34:10.542445 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:10.542378    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1c0521bb09f5a76d4c89930861a6cc5b1347e1d9ad4ac20674040f0e87e015e"
Sep 14 13:34:11.227278 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.227251    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:34:11.322852 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.322763    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-envoy\") pod \"bd99aea0-8666-4eb5-9f98-3f371211f521\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") "
Sep 14 13:34:11.322852 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.322796    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-workload-certs\") pod \"bd99aea0-8666-4eb5-9f98-3f371211f521\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") "
Sep 14 13:34:11.323071 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.322856    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-token\") pod \"bd99aea0-8666-4eb5-9f98-3f371211f521\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") "
Sep 14 13:34:11.323071 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.322877    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd99aea0-8666-4eb5-9f98-3f371211f521-istiod-ca-cert\") pod \"bd99aea0-8666-4eb5-9f98-3f371211f521\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") "
Sep 14 13:34:11.323071 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.322898    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-workload-socket\") pod \"bd99aea0-8666-4eb5-9f98-3f371211f521\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") "
Sep 14 13:34:11.323071 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.322924    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-data\") pod \"bd99aea0-8666-4eb5-9f98-3f371211f521\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") "
Sep 14 13:34:11.323071 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.322961    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-credential-socket\") pod \"bd99aea0-8666-4eb5-9f98-3f371211f521\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") "
Sep 14 13:34:11.323071 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.323011    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-podinfo\") pod \"bd99aea0-8666-4eb5-9f98-3f371211f521\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") "
Sep 14 13:34:11.323071 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.323045    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrw8f\" (UniqueName: \"kubernetes.io/projected/bd99aea0-8666-4eb5-9f98-3f371211f521-kube-api-access-mrw8f\") pod \"bd99aea0-8666-4eb5-9f98-3f371211f521\" (UID: \"bd99aea0-8666-4eb5-9f98-3f371211f521\") "
Sep 14 13:34:11.323416 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.323213    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bd99aea0-8666-4eb5-9f98-3f371211f521" (UID: "bd99aea0-8666-4eb5-9f98-3f371211f521"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:34:11.323416 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.323225    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bd99aea0-8666-4eb5-9f98-3f371211f521" (UID: "bd99aea0-8666-4eb5-9f98-3f371211f521"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:34:11.323416 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.323329    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd99aea0-8666-4eb5-9f98-3f371211f521-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bd99aea0-8666-4eb5-9f98-3f371211f521" (UID: "bd99aea0-8666-4eb5-9f98-3f371211f521"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:34:11.323567 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.323406    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bd99aea0-8666-4eb5-9f98-3f371211f521" (UID: "bd99aea0-8666-4eb5-9f98-3f371211f521"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:34:11.323567 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.323443    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-data" (OuterVolumeSpecName: "istio-data") pod "bd99aea0-8666-4eb5-9f98-3f371211f521" (UID: "bd99aea0-8666-4eb5-9f98-3f371211f521"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:34:11.323567 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.323385    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-workload-socket\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:11.323567 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.323493    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-workload-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:11.325252 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.325222    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bd99aea0-8666-4eb5-9f98-3f371211f521" (UID: "bd99aea0-8666-4eb5-9f98-3f371211f521"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 13:34:11.325355 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.325248    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd99aea0-8666-4eb5-9f98-3f371211f521-kube-api-access-mrw8f" (OuterVolumeSpecName: "kube-api-access-mrw8f") pod "bd99aea0-8666-4eb5-9f98-3f371211f521" (UID: "bd99aea0-8666-4eb5-9f98-3f371211f521"). InnerVolumeSpecName "kube-api-access-mrw8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:34:11.325460 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.325443    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-token" (OuterVolumeSpecName: "istio-token") pod "bd99aea0-8666-4eb5-9f98-3f371211f521" (UID: "bd99aea0-8666-4eb5-9f98-3f371211f521"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:34:11.325506 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.325446    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bd99aea0-8666-4eb5-9f98-3f371211f521" (UID: "bd99aea0-8666-4eb5-9f98-3f371211f521"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:34:11.424306 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.424275    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-podinfo\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:11.424306 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.424301    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrw8f\" (UniqueName: \"kubernetes.io/projected/bd99aea0-8666-4eb5-9f98-3f371211f521-kube-api-access-mrw8f\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:11.424306 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.424311    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-envoy\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:11.424535 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.424319    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-token\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:11.424535 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.424326    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd99aea0-8666-4eb5-9f98-3f371211f521-istiod-ca-cert\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:11.424535 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.424333    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-istio-data\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:11.424535 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.424341    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd99aea0-8666-4eb5-9f98-3f371211f521-credential-socket\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:11.551910 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.551880    2557 generic.go:356] "Generic (PLEG): container finished" podID="bd99aea0-8666-4eb5-9f98-3f371211f521" containerID="86b8267543d4930ec5205e920f869a2b6bdd8589baafa5a44910042dd8e1093a" exitCode=0
Sep 14 13:34:11.552069 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.551930    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"
Sep 14 13:34:11.552069 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.551969    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q" event={"ID":"bd99aea0-8666-4eb5-9f98-3f371211f521","Type":"ContainerDied","Data":"86b8267543d4930ec5205e920f869a2b6bdd8589baafa5a44910042dd8e1093a"}
Sep 14 13:34:11.552069 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.552026    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q" event={"ID":"bd99aea0-8666-4eb5-9f98-3f371211f521","Type":"ContainerDied","Data":"6c4233fac60d5521dd53f3617f24c12dc9543c123ae34d13baaf1305cbd6496b"}
Sep 14 13:34:11.552069 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.552038    2557 scope.go:117] "RemoveContainer" containerID="86b8267543d4930ec5205e920f869a2b6bdd8589baafa5a44910042dd8e1093a"
Sep 14 13:34:11.561010 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.560991    2557 scope.go:117] "RemoveContainer" containerID="86cbb1507615d12b5ecb165ae2fc217b37a38a85237a48bad013d488a934ee81"
Sep 14 13:34:11.569005 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.568988    2557 scope.go:117] "RemoveContainer" containerID="86b8267543d4930ec5205e920f869a2b6bdd8589baafa5a44910042dd8e1093a"
Sep 14 13:34:11.569245 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:34:11.569228    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86b8267543d4930ec5205e920f869a2b6bdd8589baafa5a44910042dd8e1093a\": container with ID starting with 86b8267543d4930ec5205e920f869a2b6bdd8589baafa5a44910042dd8e1093a not found: ID does not exist" containerID="86b8267543d4930ec5205e920f869a2b6bdd8589baafa5a44910042dd8e1093a"
Sep 14 13:34:11.569284 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.569258    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b8267543d4930ec5205e920f869a2b6bdd8589baafa5a44910042dd8e1093a"} err="failed to get container status \"86b8267543d4930ec5205e920f869a2b6bdd8589baafa5a44910042dd8e1093a\": rpc error: code = NotFound desc = could not find container \"86b8267543d4930ec5205e920f869a2b6bdd8589baafa5a44910042dd8e1093a\": container with ID starting with 86b8267543d4930ec5205e920f869a2b6bdd8589baafa5a44910042dd8e1093a not found: ID does not exist"
Sep 14 13:34:11.569284 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.569280    2557 scope.go:117] "RemoveContainer" containerID="86cbb1507615d12b5ecb165ae2fc217b37a38a85237a48bad013d488a934ee81"
Sep 14 13:34:11.569518 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:34:11.569500    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86cbb1507615d12b5ecb165ae2fc217b37a38a85237a48bad013d488a934ee81\": container with ID starting with 86cbb1507615d12b5ecb165ae2fc217b37a38a85237a48bad013d488a934ee81 not found: ID does not exist" containerID="86cbb1507615d12b5ecb165ae2fc217b37a38a85237a48bad013d488a934ee81"
Sep 14 13:34:11.569555 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.569526    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86cbb1507615d12b5ecb165ae2fc217b37a38a85237a48bad013d488a934ee81"} err="failed to get container status \"86cbb1507615d12b5ecb165ae2fc217b37a38a85237a48bad013d488a934ee81\": rpc error: code = NotFound desc = could not find container \"86cbb1507615d12b5ecb165ae2fc217b37a38a85237a48bad013d488a934ee81\": container with ID starting with 86cbb1507615d12b5ecb165ae2fc217b37a38a85237a48bad013d488a934ee81 not found: ID does not exist"
Sep 14 13:34:11.574197 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.574148    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"]
Sep 14 13:34:11.578068 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:11.578048    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-27ae76-openshift-default-66974cc44-x5b6q"]
Sep 14 13:34:12.049413 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:12.049382    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd99aea0-8666-4eb5-9f98-3f371211f521" path="/var/lib/kubelet/pods/bd99aea0-8666-4eb5-9f98-3f371211f521/volumes"
Sep 14 13:34:16.833403 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:16.833368    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14f3af-29823210-xbdng"]
Sep 14 13:34:16.835186 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:16.835165    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14f3af-29823210-xbdng"]
Sep 14 13:34:16.856660 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:16.856634    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw"]
Sep 14 13:34:16.856930 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:16.856899    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw" podUID="28f1ea7b-a882-4dac-8e24-82f85023c5f9" containerName="payload-processing" containerID="cri-o://0a3ac0e86faef4ae7772c4bcc77fef7586400ae7ede71925110f982228927b59" gracePeriod=30
Sep 14 13:34:17.019555 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:17.019523    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14f3af-m5whd"]
Sep 14 13:34:17.027013 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:17.026988    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14f3af-m5whd"]
Sep 14 13:34:17.107994 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:17.107970    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw"
Sep 14 13:34:17.269687 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:17.269658    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x99ld\" (UniqueName: \"kubernetes.io/projected/28f1ea7b-a882-4dac-8e24-82f85023c5f9-kube-api-access-x99ld\") pod \"28f1ea7b-a882-4dac-8e24-82f85023c5f9\" (UID: \"28f1ea7b-a882-4dac-8e24-82f85023c5f9\") "
Sep 14 13:34:17.269687 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:17.269692    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28f1ea7b-a882-4dac-8e24-82f85023c5f9-plugins-config-volume\") pod \"28f1ea7b-a882-4dac-8e24-82f85023c5f9\" (UID: \"28f1ea7b-a882-4dac-8e24-82f85023c5f9\") "
Sep 14 13:34:17.270109 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:17.270085    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f1ea7b-a882-4dac-8e24-82f85023c5f9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "28f1ea7b-a882-4dac-8e24-82f85023c5f9" (UID: "28f1ea7b-a882-4dac-8e24-82f85023c5f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:34:17.271776 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:17.271750    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f1ea7b-a882-4dac-8e24-82f85023c5f9-kube-api-access-x99ld" (OuterVolumeSpecName: "kube-api-access-x99ld") pod "28f1ea7b-a882-4dac-8e24-82f85023c5f9" (UID: "28f1ea7b-a882-4dac-8e24-82f85023c5f9"). InnerVolumeSpecName "kube-api-access-x99ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:34:17.370626 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:17.370561    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x99ld\" (UniqueName: \"kubernetes.io/projected/28f1ea7b-a882-4dac-8e24-82f85023c5f9-kube-api-access-x99ld\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:17.370626 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:17.370585    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28f1ea7b-a882-4dac-8e24-82f85023c5f9-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:17.586940 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:17.586907    2557 generic.go:356] "Generic (PLEG): container finished" podID="28f1ea7b-a882-4dac-8e24-82f85023c5f9" containerID="0a3ac0e86faef4ae7772c4bcc77fef7586400ae7ede71925110f982228927b59" exitCode=0
Sep 14 13:34:17.587099 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:17.586952    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw"
Sep 14 13:34:17.587099 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:17.586992    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw" event={"ID":"28f1ea7b-a882-4dac-8e24-82f85023c5f9","Type":"ContainerDied","Data":"0a3ac0e86faef4ae7772c4bcc77fef7586400ae7ede71925110f982228927b59"}
Sep 14 13:34:17.587099 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:17.587048    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw" event={"ID":"28f1ea7b-a882-4dac-8e24-82f85023c5f9","Type":"ContainerDied","Data":"aad11109733ecd3014b2bd62ec922c8b0ba5f475b38ca4f457e31b040ada77ff"}
Sep 14 13:34:17.587099 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:17.587080    2557 scope.go:117] "RemoveContainer" containerID="0a3ac0e86faef4ae7772c4bcc77fef7586400ae7ede71925110f982228927b59"
Sep 14 13:34:17.596134 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:17.596116    2557 scope.go:117] "RemoveContainer" containerID="0a3ac0e86faef4ae7772c4bcc77fef7586400ae7ede71925110f982228927b59"
Sep 14 13:34:17.596431 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:34:17.596411    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3ac0e86faef4ae7772c4bcc77fef7586400ae7ede71925110f982228927b59\": container with ID starting with 0a3ac0e86faef4ae7772c4bcc77fef7586400ae7ede71925110f982228927b59 not found: ID does not exist" containerID="0a3ac0e86faef4ae7772c4bcc77fef7586400ae7ede71925110f982228927b59"
Sep 14 13:34:17.596508 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:17.596452    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3ac0e86faef4ae7772c4bcc77fef7586400ae7ede71925110f982228927b59"} err="failed to get container status \"0a3ac0e86faef4ae7772c4bcc77fef7586400ae7ede71925110f982228927b59\": rpc error: code = NotFound desc = could not find container \"0a3ac0e86faef4ae7772c4bcc77fef7586400ae7ede71925110f982228927b59\": container with ID starting with 0a3ac0e86faef4ae7772c4bcc77fef7586400ae7ede71925110f982228927b59 not found: ID does not exist"
Sep 14 13:34:17.610665 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:17.610643    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw"]
Sep 14 13:34:17.616700 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:17.616678    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-14f3af-54b866949c-7zxcw"]
Sep 14 13:34:18.049163 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:18.049132    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28f1ea7b-a882-4dac-8e24-82f85023c5f9" path="/var/lib/kubelet/pods/28f1ea7b-a882-4dac-8e24-82f85023c5f9/volumes"
Sep 14 13:34:18.049519 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:18.049445    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8e99b99-a1e9-4713-b932-fa099f41b360" path="/var/lib/kubelet/pods/a8e99b99-a1e9-4713-b932-fa099f41b360/volumes"
Sep 14 13:34:18.049677 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:18.049665    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be008d5d-4051-44c1-8616-26fa3524bef0" path="/var/lib/kubelet/pods/be008d5d-4051-44c1-8616-26fa3524bef0/volumes"
Sep 14 13:34:23.741632 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:23.741599    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"]
Sep 14 13:34:23.742225 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:23.741884    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j" podUID="b2336252-c624-48ac-8aef-599b3ebabd52" containerName="istio-proxy" containerID="cri-o://ea840559eb85c589dad07863a0557b402c7b8efa90ca8002903be37b0e38257a" gracePeriod=30
Sep 14 13:34:24.095765 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:24.095676    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-eff5e8-rwklk"]
Sep 14 13:34:24.096052 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:24.096038    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8e99b99-a1e9-4713-b932-fa099f41b360" containerName="revoke-keys"
Sep 14 13:34:24.096103 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:24.096053    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8e99b99-a1e9-4713-b932-fa099f41b360" containerName="revoke-keys"
Sep 14 13:34:24.096103 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:24.096075    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd99aea0-8666-4eb5-9f98-3f371211f521" containerName="istio-proxy"
Sep 14 13:34:24.096103 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:24.096081    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd99aea0-8666-4eb5-9f98-3f371211f521" containerName="istio-proxy"
Sep 14 13:34:24.096103 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:24.096088    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28f1ea7b-a882-4dac-8e24-82f85023c5f9" containerName="payload-processing"
Sep 14 13:34:24.096103 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:24.096095    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="28f1ea7b-a882-4dac-8e24-82f85023c5f9" containerName="payload-processing"
Sep 14 13:34:24.096103 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:24.096104    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd99aea0-8666-4eb5-9f98-3f371211f521" containerName="istio-proxy"
Sep 14 13:34:24.096285 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:24.096110    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd99aea0-8666-4eb5-9f98-3f371211f521" containerName="istio-proxy"
Sep 14 13:34:24.096285 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:24.096165    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="28f1ea7b-a882-4dac-8e24-82f85023c5f9" containerName="payload-processing"
Sep 14 13:34:24.096285 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:24.096174    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd99aea0-8666-4eb5-9f98-3f371211f521" containerName="istio-proxy"
Sep 14 13:34:24.096285 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:24.096180    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8e99b99-a1e9-4713-b932-fa099f41b360" containerName="revoke-keys"
Sep 14 13:34:24.100334 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:24.100317    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-eff5e8-rwklk"
Sep 14 13:34:24.103054 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:24.103039    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbzp\""
Sep 14 13:34:24.106439 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:24.106403    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-eff5e8-rwklk"]
Sep 14 13:34:24.229381 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:24.229346    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18e805b2-8975-4242-aed9-12acb4e00284-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-eff5e8-rwklk\" (UID: \"18e805b2-8975-4242-aed9-12acb4e00284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-eff5e8-rwklk"
Sep 14 13:34:24.330782 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:24.330726    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18e805b2-8975-4242-aed9-12acb4e00284-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-eff5e8-rwklk\" (UID: \"18e805b2-8975-4242-aed9-12acb4e00284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-eff5e8-rwklk"
Sep 14 13:34:24.331315 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:24.331297    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18e805b2-8975-4242-aed9-12acb4e00284-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-eff5e8-rwklk\" (UID: \"18e805b2-8975-4242-aed9-12acb4e00284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-eff5e8-rwklk"
Sep 14 13:34:24.412481 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:24.412404    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-eff5e8-rwklk"
Sep 14 13:34:24.685326 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:24.685304    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-eff5e8-rwklk"]
Sep 14 13:34:24.687458 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:34:24.687430    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e805b2_8975_4242_aed9_12acb4e00284.slice/crio-be03d690345355d3dd9cbf6bd74cd06cb53fd4c1affb79627d9749a4b5deb8e2 WatchSource:0}: Error finding container be03d690345355d3dd9cbf6bd74cd06cb53fd4c1affb79627d9749a4b5deb8e2: Status 404 returned error can't find the container with id be03d690345355d3dd9cbf6bd74cd06cb53fd4c1affb79627d9749a4b5deb8e2
Sep 14 13:34:25.628951 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:25.628864    2557 generic.go:356] "Generic (PLEG): container finished" podID="18e805b2-8975-4242-aed9-12acb4e00284" containerID="ce0a2bd7bdbcf26c88bde985a980f5f7b42bc6dd6a51fa1c3f388c183ff6922d" exitCode=0
Sep 14 13:34:25.628951 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:25.628905    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-eff5e8-rwklk" event={"ID":"18e805b2-8975-4242-aed9-12acb4e00284","Type":"ContainerDied","Data":"ce0a2bd7bdbcf26c88bde985a980f5f7b42bc6dd6a51fa1c3f388c183ff6922d"}
Sep 14 13:34:25.629353 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:25.628963    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-eff5e8-rwklk" event={"ID":"18e805b2-8975-4242-aed9-12acb4e00284","Type":"ContainerStarted","Data":"be03d690345355d3dd9cbf6bd74cd06cb53fd4c1affb79627d9749a4b5deb8e2"}
Sep 14 13:34:26.339500 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:26.339466    2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 13:34:26.339674 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:26.339534    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j" podUID="b2336252-c624-48ac-8aef-599b3ebabd52" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 13:34:26.765721 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:26.765699    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-eff5e8-rwklk"
Sep 14 13:34:26.955648 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:26.955617    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18e805b2-8975-4242-aed9-12acb4e00284-maas-api-service-ca\") pod \"18e805b2-8975-4242-aed9-12acb4e00284\" (UID: \"18e805b2-8975-4242-aed9-12acb4e00284\") "
Sep 14 13:34:26.956019 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:26.955995    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e805b2-8975-4242-aed9-12acb4e00284-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "18e805b2-8975-4242-aed9-12acb4e00284" (UID: "18e805b2-8975-4242-aed9-12acb4e00284"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:34:27.056793 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:27.056762    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18e805b2-8975-4242-aed9-12acb4e00284-maas-api-service-ca\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:27.641392 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:27.641358    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-eff5e8-rwklk"
Sep 14 13:34:27.641571 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:27.641414    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-eff5e8-rwklk" event={"ID":"18e805b2-8975-4242-aed9-12acb4e00284","Type":"ContainerDied","Data":"be03d690345355d3dd9cbf6bd74cd06cb53fd4c1affb79627d9749a4b5deb8e2"}
Sep 14 13:34:27.641571 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:27.641464    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be03d690345355d3dd9cbf6bd74cd06cb53fd4c1affb79627d9749a4b5deb8e2"
Sep 14 13:34:28.990667 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:28.990644    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:34:29.071442 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.071374    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-workload-socket\") pod \"b2336252-c624-48ac-8aef-599b3ebabd52\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") "
Sep 14 13:34:29.071442 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.071423    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-credential-socket\") pod \"b2336252-c624-48ac-8aef-599b3ebabd52\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") "
Sep 14 13:34:29.071668 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.071451    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2336252-c624-48ac-8aef-599b3ebabd52-istio-token\") pod \"b2336252-c624-48ac-8aef-599b3ebabd52\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") "
Sep 14 13:34:29.071668 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.071471    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2336252-c624-48ac-8aef-599b3ebabd52-istio-podinfo\") pod \"b2336252-c624-48ac-8aef-599b3ebabd52\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") "
Sep 14 13:34:29.071668 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.071488    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2336252-c624-48ac-8aef-599b3ebabd52-istiod-ca-cert\") pod \"b2336252-c624-48ac-8aef-599b3ebabd52\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") "
Sep 14 13:34:29.071668 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.071512    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-istio-data\") pod \"b2336252-c624-48ac-8aef-599b3ebabd52\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") "
Sep 14 13:34:29.071668 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.071534    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktppn\" (UniqueName: \"kubernetes.io/projected/b2336252-c624-48ac-8aef-599b3ebabd52-kube-api-access-ktppn\") pod \"b2336252-c624-48ac-8aef-599b3ebabd52\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") "
Sep 14 13:34:29.071668 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.071557    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-workload-certs\") pod \"b2336252-c624-48ac-8aef-599b3ebabd52\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") "
Sep 14 13:34:29.071668 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.071589    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-istio-envoy\") pod \"b2336252-c624-48ac-8aef-599b3ebabd52\" (UID: \"b2336252-c624-48ac-8aef-599b3ebabd52\") "
Sep 14 13:34:29.072073 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.071682    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b2336252-c624-48ac-8aef-599b3ebabd52" (UID: "b2336252-c624-48ac-8aef-599b3ebabd52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:34:29.072073 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.071714    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b2336252-c624-48ac-8aef-599b3ebabd52" (UID: "b2336252-c624-48ac-8aef-599b3ebabd52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:34:29.072073 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.071862    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-istio-data" (OuterVolumeSpecName: "istio-data") pod "b2336252-c624-48ac-8aef-599b3ebabd52" (UID: "b2336252-c624-48ac-8aef-599b3ebabd52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:34:29.072073 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.071930    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-credential-socket\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:29.072073 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.071949    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-workload-socket\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:29.072073 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.071999    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b2336252-c624-48ac-8aef-599b3ebabd52" (UID: "b2336252-c624-48ac-8aef-599b3ebabd52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:34:29.072415 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.072162    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2336252-c624-48ac-8aef-599b3ebabd52-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b2336252-c624-48ac-8aef-599b3ebabd52" (UID: "b2336252-c624-48ac-8aef-599b3ebabd52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:34:29.073712 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.073680    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2336252-c624-48ac-8aef-599b3ebabd52-kube-api-access-ktppn" (OuterVolumeSpecName: "kube-api-access-ktppn") pod "b2336252-c624-48ac-8aef-599b3ebabd52" (UID: "b2336252-c624-48ac-8aef-599b3ebabd52"). InnerVolumeSpecName "kube-api-access-ktppn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:34:29.073860 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.073818    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2336252-c624-48ac-8aef-599b3ebabd52-istio-token" (OuterVolumeSpecName: "istio-token") pod "b2336252-c624-48ac-8aef-599b3ebabd52" (UID: "b2336252-c624-48ac-8aef-599b3ebabd52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:34:29.074096 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.074074    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b2336252-c624-48ac-8aef-599b3ebabd52" (UID: "b2336252-c624-48ac-8aef-599b3ebabd52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:34:29.074096 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.074080    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2336252-c624-48ac-8aef-599b3ebabd52-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b2336252-c624-48ac-8aef-599b3ebabd52" (UID: "b2336252-c624-48ac-8aef-599b3ebabd52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 13:34:29.172650 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.172617    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2336252-c624-48ac-8aef-599b3ebabd52-istio-podinfo\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:29.172650 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.172648    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2336252-c624-48ac-8aef-599b3ebabd52-istiod-ca-cert\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:29.172650 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.172657    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-istio-data\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:29.172844 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.172665    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktppn\" (UniqueName: \"kubernetes.io/projected/b2336252-c624-48ac-8aef-599b3ebabd52-kube-api-access-ktppn\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:29.172844 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.172674    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-workload-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:29.172844 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.172682    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2336252-c624-48ac-8aef-599b3ebabd52-istio-envoy\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:29.172844 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.172689    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2336252-c624-48ac-8aef-599b3ebabd52-istio-token\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:29.654216 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.654182    2557 generic.go:356] "Generic (PLEG): container finished" podID="b2336252-c624-48ac-8aef-599b3ebabd52" containerID="ea840559eb85c589dad07863a0557b402c7b8efa90ca8002903be37b0e38257a" exitCode=0
Sep 14 13:34:29.654395 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.654224    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"
Sep 14 13:34:29.654395 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.654262    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j" event={"ID":"b2336252-c624-48ac-8aef-599b3ebabd52","Type":"ContainerDied","Data":"ea840559eb85c589dad07863a0557b402c7b8efa90ca8002903be37b0e38257a"}
Sep 14 13:34:29.654395 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.654311    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j" event={"ID":"b2336252-c624-48ac-8aef-599b3ebabd52","Type":"ContainerDied","Data":"c3ac782e8d46d851ff435af0c369e5c7f784e7da91733fa6aff1d55e59f5a760"}
Sep 14 13:34:29.654395 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.654333    2557 scope.go:117] "RemoveContainer" containerID="ea840559eb85c589dad07863a0557b402c7b8efa90ca8002903be37b0e38257a"
Sep 14 13:34:29.670253 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.670234    2557 scope.go:117] "RemoveContainer" containerID="e8927b6d25dcfe302e875df825a61aabe869a1fa0c7c9d97a9be8f55ff5e585d"
Sep 14 13:34:29.678413 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.678395    2557 scope.go:117] "RemoveContainer" containerID="ea840559eb85c589dad07863a0557b402c7b8efa90ca8002903be37b0e38257a"
Sep 14 13:34:29.678671 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:34:29.678654    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea840559eb85c589dad07863a0557b402c7b8efa90ca8002903be37b0e38257a\": container with ID starting with ea840559eb85c589dad07863a0557b402c7b8efa90ca8002903be37b0e38257a not found: ID does not exist" containerID="ea840559eb85c589dad07863a0557b402c7b8efa90ca8002903be37b0e38257a"
Sep 14 13:34:29.678716 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.678680    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea840559eb85c589dad07863a0557b402c7b8efa90ca8002903be37b0e38257a"} err="failed to get container status \"ea840559eb85c589dad07863a0557b402c7b8efa90ca8002903be37b0e38257a\": rpc error: code = NotFound desc = could not find container \"ea840559eb85c589dad07863a0557b402c7b8efa90ca8002903be37b0e38257a\": container with ID starting with ea840559eb85c589dad07863a0557b402c7b8efa90ca8002903be37b0e38257a not found: ID does not exist"
Sep 14 13:34:29.678716 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.678697    2557 scope.go:117] "RemoveContainer" containerID="e8927b6d25dcfe302e875df825a61aabe869a1fa0c7c9d97a9be8f55ff5e585d"
Sep 14 13:34:29.679005 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:34:29.678988    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8927b6d25dcfe302e875df825a61aabe869a1fa0c7c9d97a9be8f55ff5e585d\": container with ID starting with e8927b6d25dcfe302e875df825a61aabe869a1fa0c7c9d97a9be8f55ff5e585d not found: ID does not exist" containerID="e8927b6d25dcfe302e875df825a61aabe869a1fa0c7c9d97a9be8f55ff5e585d"
Sep 14 13:34:29.679055 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.679008    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8927b6d25dcfe302e875df825a61aabe869a1fa0c7c9d97a9be8f55ff5e585d"} err="failed to get container status \"e8927b6d25dcfe302e875df825a61aabe869a1fa0c7c9d97a9be8f55ff5e585d\": rpc error: code = NotFound desc = could not find container \"e8927b6d25dcfe302e875df825a61aabe869a1fa0c7c9d97a9be8f55ff5e585d\": container with ID starting with e8927b6d25dcfe302e875df825a61aabe869a1fa0c7c9d97a9be8f55ff5e585d not found: ID does not exist"
Sep 14 13:34:29.681935 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.681916    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"]
Sep 14 13:34:29.686359 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:29.686340    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-14f3af-openshift-default-68c8c577c8-w6t8j"]
Sep 14 13:34:30.048875 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:30.048840    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2336252-c624-48ac-8aef-599b3ebabd52" path="/var/lib/kubelet/pods/b2336252-c624-48ac-8aef-599b3ebabd52/volumes"
Sep 14 13:34:34.834885 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:34.834856    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm"]
Sep 14 13:34:34.835380 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:34.835125    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm" podUID="8b248b33-8585-4ff5-91d0-39965c7cb281" containerName="payload-pre-processing" containerID="cri-o://bb7d11560f36c162ac0c82979016c21da029323af07c7296637b5eb96d671fb0" gracePeriod=30
Sep 14 13:34:34.969402 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:34.969368    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-eff5e8-rwklk"]
Sep 14 13:34:34.972145 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:34.972116    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-eff5e8-rwklk"]
Sep 14 13:34:35.096059 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:35.095999    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm"
Sep 14 13:34:35.121234 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:35.121207    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b248b33-8585-4ff5-91d0-39965c7cb281-plugins-config-volume\") pod \"8b248b33-8585-4ff5-91d0-39965c7cb281\" (UID: \"8b248b33-8585-4ff5-91d0-39965c7cb281\") "
Sep 14 13:34:35.121234 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:35.121236    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vgtg\" (UniqueName: \"kubernetes.io/projected/8b248b33-8585-4ff5-91d0-39965c7cb281-kube-api-access-8vgtg\") pod \"8b248b33-8585-4ff5-91d0-39965c7cb281\" (UID: \"8b248b33-8585-4ff5-91d0-39965c7cb281\") "
Sep 14 13:34:35.121617 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:35.121583    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b248b33-8585-4ff5-91d0-39965c7cb281-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b248b33-8585-4ff5-91d0-39965c7cb281" (UID: "8b248b33-8585-4ff5-91d0-39965c7cb281"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 13:34:35.123341 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:35.123318    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b248b33-8585-4ff5-91d0-39965c7cb281-kube-api-access-8vgtg" (OuterVolumeSpecName: "kube-api-access-8vgtg") pod "8b248b33-8585-4ff5-91d0-39965c7cb281" (UID: "8b248b33-8585-4ff5-91d0-39965c7cb281"). InnerVolumeSpecName "kube-api-access-8vgtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:34:35.221928 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:35.221894    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b248b33-8585-4ff5-91d0-39965c7cb281-plugins-config-volume\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:35.221928 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:35.221923    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vgtg\" (UniqueName: \"kubernetes.io/projected/8b248b33-8585-4ff5-91d0-39965c7cb281-kube-api-access-8vgtg\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:35.686282 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:35.686244    2557 generic.go:356] "Generic (PLEG): container finished" podID="8b248b33-8585-4ff5-91d0-39965c7cb281" containerID="bb7d11560f36c162ac0c82979016c21da029323af07c7296637b5eb96d671fb0" exitCode=0
Sep 14 13:34:35.686455 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:35.686288    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm"
Sep 14 13:34:35.686455 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:35.686331    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm" event={"ID":"8b248b33-8585-4ff5-91d0-39965c7cb281","Type":"ContainerDied","Data":"bb7d11560f36c162ac0c82979016c21da029323af07c7296637b5eb96d671fb0"}
Sep 14 13:34:35.686455 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:35.686380    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm" event={"ID":"8b248b33-8585-4ff5-91d0-39965c7cb281","Type":"ContainerDied","Data":"4e7229f65e5cefb07f6376a6a59421fb6bad3ce3df13cad7c8bd46990050e503"}
Sep 14 13:34:35.686455 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:35.686403    2557 scope.go:117] "RemoveContainer" containerID="bb7d11560f36c162ac0c82979016c21da029323af07c7296637b5eb96d671fb0"
Sep 14 13:34:35.695810 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:35.695791    2557 scope.go:117] "RemoveContainer" containerID="bb7d11560f36c162ac0c82979016c21da029323af07c7296637b5eb96d671fb0"
Sep 14 13:34:35.696083 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:34:35.696066    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb7d11560f36c162ac0c82979016c21da029323af07c7296637b5eb96d671fb0\": container with ID starting with bb7d11560f36c162ac0c82979016c21da029323af07c7296637b5eb96d671fb0 not found: ID does not exist" containerID="bb7d11560f36c162ac0c82979016c21da029323af07c7296637b5eb96d671fb0"
Sep 14 13:34:35.696138 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:35.696091    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb7d11560f36c162ac0c82979016c21da029323af07c7296637b5eb96d671fb0"} err="failed to get container status \"bb7d11560f36c162ac0c82979016c21da029323af07c7296637b5eb96d671fb0\": rpc error: code = NotFound desc = could not find container \"bb7d11560f36c162ac0c82979016c21da029323af07c7296637b5eb96d671fb0\": container with ID starting with bb7d11560f36c162ac0c82979016c21da029323af07c7296637b5eb96d671fb0 not found: ID does not exist"
Sep 14 13:34:35.710998 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:35.710977    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm"]
Sep 14 13:34:35.712944 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:35.712926    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-eff5e8-6c9f9b4df4-9slsm"]
Sep 14 13:34:36.048881 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:36.048848    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18e805b2-8975-4242-aed9-12acb4e00284" path="/var/lib/kubelet/pods/18e805b2-8975-4242-aed9-12acb4e00284/volumes"
Sep 14 13:34:36.049243 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:36.049196    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b248b33-8585-4ff5-91d0-39965c7cb281" path="/var/lib/kubelet/pods/8b248b33-8585-4ff5-91d0-39965c7cb281/volumes"
Sep 14 13:34:43.930530 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:43.930500    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c854ff6c6-rxllr"]
Sep 14 13:34:43.930979 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:43.930702    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-c854ff6c6-rxllr" podUID="211dcaa2-929d-44f6-8925-215ced59de6d" containerName="manager" containerID="cri-o://7e16ad01361c907c7ffa8cd59abade8597d7cb102cdaab985302c9dd1e074bf3" gracePeriod=30
Sep 14 13:34:44.171339 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:44.171316    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c854ff6c6-rxllr"
Sep 14 13:34:44.287833 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:44.287806    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/211dcaa2-929d-44f6-8925-215ced59de6d-webhook-cert\") pod \"211dcaa2-929d-44f6-8925-215ced59de6d\" (UID: \"211dcaa2-929d-44f6-8925-215ced59de6d\") "
Sep 14 13:34:44.288034 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:44.287903    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntcn9\" (UniqueName: \"kubernetes.io/projected/211dcaa2-929d-44f6-8925-215ced59de6d-kube-api-access-ntcn9\") pod \"211dcaa2-929d-44f6-8925-215ced59de6d\" (UID: \"211dcaa2-929d-44f6-8925-215ced59de6d\") "
Sep 14 13:34:44.288034 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:44.287956    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/211dcaa2-929d-44f6-8925-215ced59de6d-metrics-certs\") pod \"211dcaa2-929d-44f6-8925-215ced59de6d\" (UID: \"211dcaa2-929d-44f6-8925-215ced59de6d\") "
Sep 14 13:34:44.289882 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:44.289848    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/211dcaa2-929d-44f6-8925-215ced59de6d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "211dcaa2-929d-44f6-8925-215ced59de6d" (UID: "211dcaa2-929d-44f6-8925-215ced59de6d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:34:44.289986 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:44.289904    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/211dcaa2-929d-44f6-8925-215ced59de6d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "211dcaa2-929d-44f6-8925-215ced59de6d" (UID: "211dcaa2-929d-44f6-8925-215ced59de6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:34:44.289986 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:44.289957    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/211dcaa2-929d-44f6-8925-215ced59de6d-kube-api-access-ntcn9" (OuterVolumeSpecName: "kube-api-access-ntcn9") pod "211dcaa2-929d-44f6-8925-215ced59de6d" (UID: "211dcaa2-929d-44f6-8925-215ced59de6d"). InnerVolumeSpecName "kube-api-access-ntcn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:34:44.388971 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:44.388941    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntcn9\" (UniqueName: \"kubernetes.io/projected/211dcaa2-929d-44f6-8925-215ced59de6d-kube-api-access-ntcn9\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:44.388971 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:44.388967    2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/211dcaa2-929d-44f6-8925-215ced59de6d-metrics-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:44.388971 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:44.388976    2557 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/211dcaa2-929d-44f6-8925-215ced59de6d-webhook-cert\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:34:44.734083 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:44.734051    2557 generic.go:356] "Generic (PLEG): container finished" podID="211dcaa2-929d-44f6-8925-215ced59de6d" containerID="7e16ad01361c907c7ffa8cd59abade8597d7cb102cdaab985302c9dd1e074bf3" exitCode=0
Sep 14 13:34:44.734228 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:44.734091    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c854ff6c6-rxllr"
Sep 14 13:34:44.734228 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:44.734179    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c854ff6c6-rxllr" event={"ID":"211dcaa2-929d-44f6-8925-215ced59de6d","Type":"ContainerDied","Data":"7e16ad01361c907c7ffa8cd59abade8597d7cb102cdaab985302c9dd1e074bf3"}
Sep 14 13:34:44.734228 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:44.734226    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c854ff6c6-rxllr" event={"ID":"211dcaa2-929d-44f6-8925-215ced59de6d","Type":"ContainerDied","Data":"829e23cc8a957204ae7143d75e773dfee9633df6fefdd3a221b6197fc532a808"}
Sep 14 13:34:44.734336 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:44.734255    2557 scope.go:117] "RemoveContainer" containerID="7e16ad01361c907c7ffa8cd59abade8597d7cb102cdaab985302c9dd1e074bf3"
Sep 14 13:34:44.744835 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:44.744818    2557 scope.go:117] "RemoveContainer" containerID="7e16ad01361c907c7ffa8cd59abade8597d7cb102cdaab985302c9dd1e074bf3"
Sep 14 13:34:44.745100 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:34:44.745082    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e16ad01361c907c7ffa8cd59abade8597d7cb102cdaab985302c9dd1e074bf3\": container with ID starting with 7e16ad01361c907c7ffa8cd59abade8597d7cb102cdaab985302c9dd1e074bf3 not found: ID does not exist" containerID="7e16ad01361c907c7ffa8cd59abade8597d7cb102cdaab985302c9dd1e074bf3"
Sep 14 13:34:44.745169 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:44.745106    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e16ad01361c907c7ffa8cd59abade8597d7cb102cdaab985302c9dd1e074bf3"} err="failed to get container status \"7e16ad01361c907c7ffa8cd59abade8597d7cb102cdaab985302c9dd1e074bf3\": rpc error: code = NotFound desc = could not find container \"7e16ad01361c907c7ffa8cd59abade8597d7cb102cdaab985302c9dd1e074bf3\": container with ID starting with 7e16ad01361c907c7ffa8cd59abade8597d7cb102cdaab985302c9dd1e074bf3 not found: ID does not exist"
Sep 14 13:34:44.756048 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:44.756010    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c854ff6c6-rxllr"]
Sep 14 13:34:44.760053 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:44.760033    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c854ff6c6-rxllr"]
Sep 14 13:34:46.048676 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:34:46.048647    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="211dcaa2-929d-44f6-8925-215ced59de6d" path="/var/lib/kubelet/pods/211dcaa2-929d-44f6-8925-215ced59de6d/volumes"
Sep 14 13:35:02.693429 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:02.693397    2557 scope.go:117] "RemoveContainer" containerID="e16d76db44fab349ef16a4f85cebfd76d61c131941af9812459a24577443152b"
Sep 14 13:35:02.701515 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:02.701499    2557 scope.go:117] "RemoveContainer" containerID="838113c53429f884ff7d0dc24b2fd1cd393684442ed4dbebe630b961379b5a7c"
Sep 14 13:35:45.534706 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.534672    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c854ff6c6-h7fck"]
Sep 14 13:35:45.535185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.534993    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="211dcaa2-929d-44f6-8925-215ced59de6d" containerName="manager"
Sep 14 13:35:45.535185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.535005    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="211dcaa2-929d-44f6-8925-215ced59de6d" containerName="manager"
Sep 14 13:35:45.535185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.535014    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b248b33-8585-4ff5-91d0-39965c7cb281" containerName="payload-pre-processing"
Sep 14 13:35:45.535185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.535020    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b248b33-8585-4ff5-91d0-39965c7cb281" containerName="payload-pre-processing"
Sep 14 13:35:45.535185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.535029    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2336252-c624-48ac-8aef-599b3ebabd52" containerName="istio-proxy"
Sep 14 13:35:45.535185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.535036    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2336252-c624-48ac-8aef-599b3ebabd52" containerName="istio-proxy"
Sep 14 13:35:45.535185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.535048    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18e805b2-8975-4242-aed9-12acb4e00284" containerName="revoke-keys"
Sep 14 13:35:45.535185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.535053    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="18e805b2-8975-4242-aed9-12acb4e00284" containerName="revoke-keys"
Sep 14 13:35:45.535185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.535106    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2336252-c624-48ac-8aef-599b3ebabd52" containerName="istio-proxy"
Sep 14 13:35:45.535185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.535113    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2336252-c624-48ac-8aef-599b3ebabd52" containerName="istio-proxy"
Sep 14 13:35:45.535185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.535121    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd99aea0-8666-4eb5-9f98-3f371211f521" containerName="istio-proxy"
Sep 14 13:35:45.535185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.535126    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="18e805b2-8975-4242-aed9-12acb4e00284" containerName="revoke-keys"
Sep 14 13:35:45.535185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.535132    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="211dcaa2-929d-44f6-8925-215ced59de6d" containerName="manager"
Sep 14 13:35:45.535185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.535138    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b248b33-8585-4ff5-91d0-39965c7cb281" containerName="payload-pre-processing"
Sep 14 13:35:45.538111 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.538094    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c854ff6c6-h7fck"
Sep 14 13:35:45.541995 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.541973    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gbc6v\""
Sep 14 13:35:45.542109 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.541973    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 14 13:35:45.542109 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.542016    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 14 13:35:45.542109 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.542034    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 14 13:35:45.548182 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.548137    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c854ff6c6-h7fck"]
Sep 14 13:35:45.648375 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.648353    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af232a4d-46ad-474b-9c5e-13a8e68e982a-webhook-cert\") pod \"maas-controller-c854ff6c6-h7fck\" (UID: \"af232a4d-46ad-474b-9c5e-13a8e68e982a\") " pod="opendatahub/maas-controller-c854ff6c6-h7fck"
Sep 14 13:35:45.648488 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.648380    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjpjd\" (UniqueName: \"kubernetes.io/projected/af232a4d-46ad-474b-9c5e-13a8e68e982a-kube-api-access-hjpjd\") pod \"maas-controller-c854ff6c6-h7fck\" (UID: \"af232a4d-46ad-474b-9c5e-13a8e68e982a\") " pod="opendatahub/maas-controller-c854ff6c6-h7fck"
Sep 14 13:35:45.648488 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.648423    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af232a4d-46ad-474b-9c5e-13a8e68e982a-metrics-certs\") pod \"maas-controller-c854ff6c6-h7fck\" (UID: \"af232a4d-46ad-474b-9c5e-13a8e68e982a\") " pod="opendatahub/maas-controller-c854ff6c6-h7fck"
Sep 14 13:35:45.749083 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.749055    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af232a4d-46ad-474b-9c5e-13a8e68e982a-metrics-certs\") pod \"maas-controller-c854ff6c6-h7fck\" (UID: \"af232a4d-46ad-474b-9c5e-13a8e68e982a\") " pod="opendatahub/maas-controller-c854ff6c6-h7fck"
Sep 14 13:35:45.749195 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.749135    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af232a4d-46ad-474b-9c5e-13a8e68e982a-webhook-cert\") pod \"maas-controller-c854ff6c6-h7fck\" (UID: \"af232a4d-46ad-474b-9c5e-13a8e68e982a\") " pod="opendatahub/maas-controller-c854ff6c6-h7fck"
Sep 14 13:35:45.749195 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.749163    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjpjd\" (UniqueName: \"kubernetes.io/projected/af232a4d-46ad-474b-9c5e-13a8e68e982a-kube-api-access-hjpjd\") pod \"maas-controller-c854ff6c6-h7fck\" (UID: \"af232a4d-46ad-474b-9c5e-13a8e68e982a\") " pod="opendatahub/maas-controller-c854ff6c6-h7fck"
Sep 14 13:35:45.751499 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.751478    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af232a4d-46ad-474b-9c5e-13a8e68e982a-metrics-certs\") pod \"maas-controller-c854ff6c6-h7fck\" (UID: \"af232a4d-46ad-474b-9c5e-13a8e68e982a\") " pod="opendatahub/maas-controller-c854ff6c6-h7fck"
Sep 14 13:35:45.751685 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.751661    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af232a4d-46ad-474b-9c5e-13a8e68e982a-webhook-cert\") pod \"maas-controller-c854ff6c6-h7fck\" (UID: \"af232a4d-46ad-474b-9c5e-13a8e68e982a\") " pod="opendatahub/maas-controller-c854ff6c6-h7fck"
Sep 14 13:35:45.757984 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.757946    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjpjd\" (UniqueName: \"kubernetes.io/projected/af232a4d-46ad-474b-9c5e-13a8e68e982a-kube-api-access-hjpjd\") pod \"maas-controller-c854ff6c6-h7fck\" (UID: \"af232a4d-46ad-474b-9c5e-13a8e68e982a\") " pod="opendatahub/maas-controller-c854ff6c6-h7fck"
Sep 14 13:35:45.849268 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:45.849210    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c854ff6c6-h7fck"
Sep 14 13:35:46.121126 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:46.121103    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c854ff6c6-h7fck"]
Sep 14 13:35:46.123364 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:35:46.123337    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf232a4d_46ad_474b_9c5e_13a8e68e982a.slice/crio-2ee9c438f52df51ff336455ed259106ee3df587cf3d0114ee619a2638eadc669 WatchSource:0}: Error finding container 2ee9c438f52df51ff336455ed259106ee3df587cf3d0114ee619a2638eadc669: Status 404 returned error can't find the container with id 2ee9c438f52df51ff336455ed259106ee3df587cf3d0114ee619a2638eadc669
Sep 14 13:35:46.124606 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:46.124589    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 13:35:47.062767 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:47.062721    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c854ff6c6-h7fck" event={"ID":"af232a4d-46ad-474b-9c5e-13a8e68e982a","Type":"ContainerStarted","Data":"c14bf478619a91562db4af5326b7471cf178c78bb97ae163960d54866a51630d"}
Sep 14 13:35:47.063141 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:47.062785    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c854ff6c6-h7fck" event={"ID":"af232a4d-46ad-474b-9c5e-13a8e68e982a","Type":"ContainerStarted","Data":"2ee9c438f52df51ff336455ed259106ee3df587cf3d0114ee619a2638eadc669"}
Sep 14 13:35:47.063141 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:47.062815    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c854ff6c6-h7fck"
Sep 14 13:35:47.081572 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:47.081522    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c854ff6c6-h7fck" podStartSLOduration=1.6131913199999999 podStartE2EDuration="2.081506801s" totalImagesPullingTime="468.315481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:35:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:35:47.08014475 +0000 UTC m=+2265.514820270" watchObservedRunningTime="2026-09-14 13:35:47.081506801 +0000 UTC m=+2265.516182320"
Sep 14 13:35:58.070348 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:35:58.070321    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c854ff6c6-h7fck"
Sep 14 13:36:01.131256 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:01.131163    2557 generic.go:356] "Generic (PLEG): container finished" podID="6fd5deac-72f2-4b25-9db8-82735b3ad959" containerID="62181b74425d6452d2f81aa4bb74a65dbcb3f286fbf6ea220df78dcd66df5b0a" exitCode=1
Sep 14 13:36:01.131256 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:01.131242    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gmncm" event={"ID":"6fd5deac-72f2-4b25-9db8-82735b3ad959","Type":"ContainerDied","Data":"62181b74425d6452d2f81aa4bb74a65dbcb3f286fbf6ea220df78dcd66df5b0a"}
Sep 14 13:36:01.131699 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:01.131542    2557 scope.go:117] "RemoveContainer" containerID="62181b74425d6452d2f81aa4bb74a65dbcb3f286fbf6ea220df78dcd66df5b0a"
Sep 14 13:36:02.138233 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:02.138149    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gmncm" event={"ID":"6fd5deac-72f2-4b25-9db8-82735b3ad959","Type":"ContainerStarted","Data":"5622f7624f6cd9ac02490e536a1993196c4fac85a3819c9d4655e7d227fccd25"}
Sep 14 13:36:02.138652 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:02.138359    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gmncm"
Sep 14 13:36:02.740620 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:02.740592    2557 scope.go:117] "RemoveContainer" containerID="5d2d7cde776ba5f0c74a90a0d39fdb36b76426b076cf0adfc6a8653500864f20"
Sep 14 13:36:02.748899 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:02.748877    2557 scope.go:117] "RemoveContainer" containerID="3f42c1f326b3fd593e6a0e0d10c610bf6c0c2cd503d57ba3ce643c4fecb4e05b"
Sep 14 13:36:02.756191 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:02.756157    2557 scope.go:117] "RemoveContainer" containerID="8d2eaca4f63b8033ad3f8d3ccc0fb5783f9750c0a346a8afd20872a0ee1827af"
Sep 14 13:36:18.398189 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:18.398153    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c854ff6c6-h7fck"]
Sep 14 13:36:18.400924 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:18.398378    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-c854ff6c6-h7fck" podUID="af232a4d-46ad-474b-9c5e-13a8e68e982a" containerName="manager" containerID="cri-o://c14bf478619a91562db4af5326b7471cf178c78bb97ae163960d54866a51630d" gracePeriod=30
Sep 14 13:36:18.644286 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:18.644261    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c854ff6c6-h7fck"
Sep 14 13:36:18.683685 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:18.683618    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af232a4d-46ad-474b-9c5e-13a8e68e982a-metrics-certs\") pod \"af232a4d-46ad-474b-9c5e-13a8e68e982a\" (UID: \"af232a4d-46ad-474b-9c5e-13a8e68e982a\") "
Sep 14 13:36:18.683685 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:18.683654    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af232a4d-46ad-474b-9c5e-13a8e68e982a-webhook-cert\") pod \"af232a4d-46ad-474b-9c5e-13a8e68e982a\" (UID: \"af232a4d-46ad-474b-9c5e-13a8e68e982a\") "
Sep 14 13:36:18.683685 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:18.683684    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjpjd\" (UniqueName: \"kubernetes.io/projected/af232a4d-46ad-474b-9c5e-13a8e68e982a-kube-api-access-hjpjd\") pod \"af232a4d-46ad-474b-9c5e-13a8e68e982a\" (UID: \"af232a4d-46ad-474b-9c5e-13a8e68e982a\") "
Sep 14 13:36:18.685725 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:18.685692    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af232a4d-46ad-474b-9c5e-13a8e68e982a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "af232a4d-46ad-474b-9c5e-13a8e68e982a" (UID: "af232a4d-46ad-474b-9c5e-13a8e68e982a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:36:18.685910 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:18.685727    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af232a4d-46ad-474b-9c5e-13a8e68e982a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "af232a4d-46ad-474b-9c5e-13a8e68e982a" (UID: "af232a4d-46ad-474b-9c5e-13a8e68e982a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:36:18.685910 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:18.685875    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af232a4d-46ad-474b-9c5e-13a8e68e982a-kube-api-access-hjpjd" (OuterVolumeSpecName: "kube-api-access-hjpjd") pod "af232a4d-46ad-474b-9c5e-13a8e68e982a" (UID: "af232a4d-46ad-474b-9c5e-13a8e68e982a"). InnerVolumeSpecName "kube-api-access-hjpjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:36:18.784427 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:18.784391    2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af232a4d-46ad-474b-9c5e-13a8e68e982a-metrics-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:36:18.784427 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:18.784422    2557 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af232a4d-46ad-474b-9c5e-13a8e68e982a-webhook-cert\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:36:18.784604 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:18.784435    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjpjd\" (UniqueName: \"kubernetes.io/projected/af232a4d-46ad-474b-9c5e-13a8e68e982a-kube-api-access-hjpjd\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:36:19.224116 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:19.224083    2557 generic.go:356] "Generic (PLEG): container finished" podID="af232a4d-46ad-474b-9c5e-13a8e68e982a" containerID="c14bf478619a91562db4af5326b7471cf178c78bb97ae163960d54866a51630d" exitCode=0
Sep 14 13:36:19.224304 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:19.224132    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c854ff6c6-h7fck"
Sep 14 13:36:19.224304 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:19.224178    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c854ff6c6-h7fck" event={"ID":"af232a4d-46ad-474b-9c5e-13a8e68e982a","Type":"ContainerDied","Data":"c14bf478619a91562db4af5326b7471cf178c78bb97ae163960d54866a51630d"}
Sep 14 13:36:19.224304 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:19.224234    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c854ff6c6-h7fck" event={"ID":"af232a4d-46ad-474b-9c5e-13a8e68e982a","Type":"ContainerDied","Data":"2ee9c438f52df51ff336455ed259106ee3df587cf3d0114ee619a2638eadc669"}
Sep 14 13:36:19.224304 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:19.224262    2557 scope.go:117] "RemoveContainer" containerID="c14bf478619a91562db4af5326b7471cf178c78bb97ae163960d54866a51630d"
Sep 14 13:36:19.233294 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:19.233279    2557 scope.go:117] "RemoveContainer" containerID="c14bf478619a91562db4af5326b7471cf178c78bb97ae163960d54866a51630d"
Sep 14 13:36:19.233543 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:36:19.233523    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c14bf478619a91562db4af5326b7471cf178c78bb97ae163960d54866a51630d\": container with ID starting with c14bf478619a91562db4af5326b7471cf178c78bb97ae163960d54866a51630d not found: ID does not exist" containerID="c14bf478619a91562db4af5326b7471cf178c78bb97ae163960d54866a51630d"
Sep 14 13:36:19.233616 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:19.233550    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c14bf478619a91562db4af5326b7471cf178c78bb97ae163960d54866a51630d"} err="failed to get container status \"c14bf478619a91562db4af5326b7471cf178c78bb97ae163960d54866a51630d\": rpc error: code = NotFound desc = could not find container \"c14bf478619a91562db4af5326b7471cf178c78bb97ae163960d54866a51630d\": container with ID starting with c14bf478619a91562db4af5326b7471cf178c78bb97ae163960d54866a51630d not found: ID does not exist"
Sep 14 13:36:19.244798 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:19.244778    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c854ff6c6-h7fck"]
Sep 14 13:36:19.247904 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:19.247885    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c854ff6c6-h7fck"]
Sep 14 13:36:20.023016 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.022983    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c854ff6c6-glrk4"]
Sep 14 13:36:20.023376 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.023302    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2336252-c624-48ac-8aef-599b3ebabd52" containerName="istio-proxy"
Sep 14 13:36:20.023376 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.023313    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2336252-c624-48ac-8aef-599b3ebabd52" containerName="istio-proxy"
Sep 14 13:36:20.023376 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.023328    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af232a4d-46ad-474b-9c5e-13a8e68e982a" containerName="manager"
Sep 14 13:36:20.023376 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.023334    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="af232a4d-46ad-474b-9c5e-13a8e68e982a" containerName="manager"
Sep 14 13:36:20.023513 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.023391    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="af232a4d-46ad-474b-9c5e-13a8e68e982a" containerName="manager"
Sep 14 13:36:20.027730 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.027711    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c854ff6c6-glrk4"
Sep 14 13:36:20.031986 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.031957    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gbc6v\""
Sep 14 13:36:20.032117 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.032000    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 14 13:36:20.032117 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.032035    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 14 13:36:20.032117 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.032078    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 14 13:36:20.034814 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.034451    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c854ff6c6-glrk4"]
Sep 14 13:36:20.048720 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.048697    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af232a4d-46ad-474b-9c5e-13a8e68e982a" path="/var/lib/kubelet/pods/af232a4d-46ad-474b-9c5e-13a8e68e982a/volumes"
Sep 14 13:36:20.094293 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.094269    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ktxh\" (UniqueName: \"kubernetes.io/projected/16d650af-bc70-4253-8e90-c270a7911fcc-kube-api-access-9ktxh\") pod \"maas-controller-c854ff6c6-glrk4\" (UID: \"16d650af-bc70-4253-8e90-c270a7911fcc\") " pod="opendatahub/maas-controller-c854ff6c6-glrk4"
Sep 14 13:36:20.094387 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.094335    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16d650af-bc70-4253-8e90-c270a7911fcc-metrics-certs\") pod \"maas-controller-c854ff6c6-glrk4\" (UID: \"16d650af-bc70-4253-8e90-c270a7911fcc\") " pod="opendatahub/maas-controller-c854ff6c6-glrk4"
Sep 14 13:36:20.094443 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.094392    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16d650af-bc70-4253-8e90-c270a7911fcc-webhook-cert\") pod \"maas-controller-c854ff6c6-glrk4\" (UID: \"16d650af-bc70-4253-8e90-c270a7911fcc\") " pod="opendatahub/maas-controller-c854ff6c6-glrk4"
Sep 14 13:36:20.195568 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.195543    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ktxh\" (UniqueName: \"kubernetes.io/projected/16d650af-bc70-4253-8e90-c270a7911fcc-kube-api-access-9ktxh\") pod \"maas-controller-c854ff6c6-glrk4\" (UID: \"16d650af-bc70-4253-8e90-c270a7911fcc\") " pod="opendatahub/maas-controller-c854ff6c6-glrk4"
Sep 14 13:36:20.195659 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.195586    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16d650af-bc70-4253-8e90-c270a7911fcc-metrics-certs\") pod \"maas-controller-c854ff6c6-glrk4\" (UID: \"16d650af-bc70-4253-8e90-c270a7911fcc\") " pod="opendatahub/maas-controller-c854ff6c6-glrk4"
Sep 14 13:36:20.195659 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.195609    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16d650af-bc70-4253-8e90-c270a7911fcc-webhook-cert\") pod \"maas-controller-c854ff6c6-glrk4\" (UID: \"16d650af-bc70-4253-8e90-c270a7911fcc\") " pod="opendatahub/maas-controller-c854ff6c6-glrk4"
Sep 14 13:36:20.198185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.198162    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16d650af-bc70-4253-8e90-c270a7911fcc-webhook-cert\") pod \"maas-controller-c854ff6c6-glrk4\" (UID: \"16d650af-bc70-4253-8e90-c270a7911fcc\") " pod="opendatahub/maas-controller-c854ff6c6-glrk4"
Sep 14 13:36:20.198273 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.198221    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16d650af-bc70-4253-8e90-c270a7911fcc-metrics-certs\") pod \"maas-controller-c854ff6c6-glrk4\" (UID: \"16d650af-bc70-4253-8e90-c270a7911fcc\") " pod="opendatahub/maas-controller-c854ff6c6-glrk4"
Sep 14 13:36:20.203729 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.203709    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ktxh\" (UniqueName: \"kubernetes.io/projected/16d650af-bc70-4253-8e90-c270a7911fcc-kube-api-access-9ktxh\") pod \"maas-controller-c854ff6c6-glrk4\" (UID: \"16d650af-bc70-4253-8e90-c270a7911fcc\") " pod="opendatahub/maas-controller-c854ff6c6-glrk4"
Sep 14 13:36:20.338835 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.338774    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c854ff6c6-glrk4"
Sep 14 13:36:20.613000 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:20.612897    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c854ff6c6-glrk4"]
Sep 14 13:36:20.615852 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:36:20.615824    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d650af_bc70_4253_8e90_c270a7911fcc.slice/crio-d174ae3857186c6c565445a27d57eb89f4a0d0479f6219392feb78cc62372951 WatchSource:0}: Error finding container d174ae3857186c6c565445a27d57eb89f4a0d0479f6219392feb78cc62372951: Status 404 returned error can't find the container with id d174ae3857186c6c565445a27d57eb89f4a0d0479f6219392feb78cc62372951
Sep 14 13:36:21.237609 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:21.237580    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c854ff6c6-glrk4" event={"ID":"16d650af-bc70-4253-8e90-c270a7911fcc","Type":"ContainerStarted","Data":"5c57aef2eae9221de470ade724997d8d04e142d5133ab6d5bb005df97ca278e8"}
Sep 14 13:36:21.238014 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:21.237634    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c854ff6c6-glrk4" event={"ID":"16d650af-bc70-4253-8e90-c270a7911fcc","Type":"ContainerStarted","Data":"d174ae3857186c6c565445a27d57eb89f4a0d0479f6219392feb78cc62372951"}
Sep 14 13:36:21.238014 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:21.237657    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c854ff6c6-glrk4"
Sep 14 13:36:21.256830 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:21.256787    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c854ff6c6-glrk4" podStartSLOduration=0.847679519 podStartE2EDuration="1.256770966s" totalImagesPullingTime="409.091447ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:36:21.255498667 +0000 UTC m=+2299.690174186" watchObservedRunningTime="2026-09-14 13:36:21.256770966 +0000 UTC m=+2299.691446485"
Sep 14 13:36:32.245479 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:32.245448    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c854ff6c6-glrk4"
Sep 14 13:36:33.147704 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:36:33.147675    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gmncm"
Sep 14 13:38:02.252084 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:38:02.252055    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwqnf_62d953d6-3b86-4c95-9a47-9579d88993a7/ovn-acl-logging/0.log"
Sep 14 13:38:02.254522 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:38:02.254500    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwqnf_62d953d6-3b86-4c95-9a47-9579d88993a7/ovn-acl-logging/0.log"
Sep 14 13:38:28.362879 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:38:28.362790    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-b97dp" podUID="e8eda6eb-1097-4a02-b60a-45eb7ddcc87e" containerName="konnectivity-agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 13:38:28.363478 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:38:28.362909    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b97dp"
Sep 14 13:38:28.363726 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:38:28.363701    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-b97dp" podUID="e8eda6eb-1097-4a02-b60a-45eb7ddcc87e" containerName="konnectivity-agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 13:38:58.364832 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:38:58.364791    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b97dp"
Sep 14 13:39:02.797895 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:02.797864    2557 scope.go:117] "RemoveContainer" containerID="c656f35578b1abeb19f78448fc1878bb1b1bd31140a68949f019ed456fa0499e"
Sep 14 13:39:02.806202 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:02.806181    2557 scope.go:117] "RemoveContainer" containerID="9a9c590bf813876552a50abcc9d4b0ddcff973d7c71df2a91df4c5292258885e"
Sep 14 13:39:14.405944 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.405910    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"]
Sep 14 13:39:14.409504 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.409487    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:14.412092 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.412068    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-ff422ea2-kserve-self-signed-certs\""
Sep 14 13:39:14.417577 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.417556    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"]
Sep 14 13:39:14.506754 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.506714    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-dshm\") pod \"e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") " pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:14.506916 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.506787    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-model-cache\") pod \"e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") " pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:14.506916 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.506837    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24bm8\" (UniqueName: \"kubernetes.io/projected/71102503-87cb-41cb-b452-61c418cd9f30-kube-api-access-24bm8\") pod \"e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") " pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:14.507071 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.506948    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-home\") pod \"e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") " pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:14.507071 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.507010    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-tmp-dir\") pod \"e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") " pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:14.507071 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.507044    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71102503-87cb-41cb-b452-61c418cd9f30-tls-certs\") pod \"e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") " pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:14.608115 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.608082    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-tmp-dir\") pod \"e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") " pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:14.608257 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.608127    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71102503-87cb-41cb-b452-61c418cd9f30-tls-certs\") pod \"e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") " pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:14.608257 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.608193    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-dshm\") pod \"e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") " pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:14.608257 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.608237    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-model-cache\") pod \"e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") " pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:14.608424 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.608286    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24bm8\" (UniqueName: \"kubernetes.io/projected/71102503-87cb-41cb-b452-61c418cd9f30-kube-api-access-24bm8\") pod \"e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") " pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:14.608424 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.608344    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-home\") pod \"e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") " pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:14.608785 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.608714    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-tmp-dir\") pod \"e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") " pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:14.608785 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.608777    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-home\") pod \"e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") " pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:14.608942 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.608808    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-model-cache\") pod \"e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") " pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:14.610452 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.610431    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-dshm\") pod \"e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") " pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:14.610961 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.610944    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71102503-87cb-41cb-b452-61c418cd9f30-tls-certs\") pod \"e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") " pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:14.616704 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.616676    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24bm8\" (UniqueName: \"kubernetes.io/projected/71102503-87cb-41cb-b452-61c418cd9f30-kube-api-access-24bm8\") pod \"e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") " pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:14.718630 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.718608    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:14.991676 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:14.991651    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"]
Sep 14 13:39:14.994085 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:39:14.994043    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71102503_87cb_41cb_b452_61c418cd9f30.slice/crio-0c4c65807228e5bea24ac06ba89fa3aefebe86b1c11c620c6715830f3f727449 WatchSource:0}: Error finding container 0c4c65807228e5bea24ac06ba89fa3aefebe86b1c11c620c6715830f3f727449: Status 404 returned error can't find the container with id 0c4c65807228e5bea24ac06ba89fa3aefebe86b1c11c620c6715830f3f727449
Sep 14 13:39:15.092335 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:15.092308    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l" event={"ID":"71102503-87cb-41cb-b452-61c418cd9f30","Type":"ContainerStarted","Data":"963016c236e2adbecca6c2b6a4291ff99d892b5d3be132c5a4054c8493d5ed0b"}
Sep 14 13:39:15.092494 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:15.092345    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l" event={"ID":"71102503-87cb-41cb-b452-61c418cd9f30","Type":"ContainerStarted","Data":"0c4c65807228e5bea24ac06ba89fa3aefebe86b1c11c620c6715830f3f727449"}
Sep 14 13:39:15.113631 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:15.113584    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l" podStartSLOduration=1.113569088 podStartE2EDuration="1.113569088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:39:15.111777034 +0000 UTC m=+2473.546452544" watchObservedRunningTime="2026-09-14 13:39:15.113569088 +0000 UTC m=+2473.548244604"
Sep 14 13:39:24.718927 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:24.718890    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:24.719377 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:24.718939    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:24.731178 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:24.731152    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:25.156103 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:25.156022    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:39:32.900128 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:32.900099    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"]
Sep 14 13:39:32.904768 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:32.904734    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:32.907276 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:32.907254    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-ff422ea2-kserve-self-signed-certs\""
Sep 14 13:39:32.911401 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:32.911377    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"]
Sep 14 13:39:33.054894 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:33.054864    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c13521d-e180-484e-b731-2adacb12c86c-tls-certs\") pod \"e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") " pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:33.055060 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:33.054911    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-dshm\") pod \"e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") " pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:33.055060 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:33.054942    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-model-cache\") pod \"e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") " pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:33.055060 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:33.054991    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-tmp-dir\") pod \"e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") " pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:33.055060 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:33.055021    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq4m5\" (UniqueName: \"kubernetes.io/projected/7c13521d-e180-484e-b731-2adacb12c86c-kube-api-access-dq4m5\") pod \"e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") " pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:33.055060 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:33.055048    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-home\") pod \"e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") " pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:33.155760 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:33.155667    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-dshm\") pod \"e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") " pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:33.155760 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:33.155702    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-model-cache\") pod \"e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") " pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:33.155760 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:33.155721    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-tmp-dir\") pod \"e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") " pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:33.155760 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:33.155757    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq4m5\" (UniqueName: \"kubernetes.io/projected/7c13521d-e180-484e-b731-2adacb12c86c-kube-api-access-dq4m5\") pod \"e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") " pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:33.156029 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:33.155813    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-home\") pod \"e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") " pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:33.156029 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:33.155899    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c13521d-e180-484e-b731-2adacb12c86c-tls-certs\") pod \"e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") " pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:33.156197 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:33.156177    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-model-cache\") pod \"e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") " pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:33.156265 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:33.156200    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-tmp-dir\") pod \"e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") " pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:33.156265 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:33.156238    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-home\") pod \"e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") " pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:33.158002 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:33.157976    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-dshm\") pod \"e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") " pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:33.158374 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:33.158348    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c13521d-e180-484e-b731-2adacb12c86c-tls-certs\") pod \"e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") " pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:33.168877 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:33.168854    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq4m5\" (UniqueName: \"kubernetes.io/projected/7c13521d-e180-484e-b731-2adacb12c86c-kube-api-access-dq4m5\") pod \"e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") " pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:33.216965 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:33.216934    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:33.490869 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:33.490840    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"]
Sep 14 13:39:33.494078 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:39:33.494053    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c13521d_e180_484e_b731_2adacb12c86c.slice/crio-ba35c3de8d35dce5d2df6952b626134887eea73cd141cdf4702b5be3b26c4694 WatchSource:0}: Error finding container ba35c3de8d35dce5d2df6952b626134887eea73cd141cdf4702b5be3b26c4694: Status 404 returned error can't find the container with id ba35c3de8d35dce5d2df6952b626134887eea73cd141cdf4702b5be3b26c4694
Sep 14 13:39:34.186288 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:34.186257    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8" event={"ID":"7c13521d-e180-484e-b731-2adacb12c86c","Type":"ContainerStarted","Data":"926744f1ccd6c99287cee78e65a050d2e7833470fb073b30894628cbfc885bbb"}
Sep 14 13:39:34.186698 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:34.186296    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8" event={"ID":"7c13521d-e180-484e-b731-2adacb12c86c","Type":"ContainerStarted","Data":"ba35c3de8d35dce5d2df6952b626134887eea73cd141cdf4702b5be3b26c4694"}
Sep 14 13:39:34.203261 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:34.203203    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8" podStartSLOduration=2.203185151 podStartE2EDuration="2.203185151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:39:34.200943516 +0000 UTC m=+2492.635619036" watchObservedRunningTime="2026-09-14 13:39:34.203185151 +0000 UTC m=+2492.637860672"
Sep 14 13:39:43.217725 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:43.217693    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:43.218203 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:43.217734    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:43.230330 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:43.230302    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:39:43.241302 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:39:43.241285    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:40:02.827603 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:02.827531    2557 scope.go:117] "RemoveContainer" containerID="25a096f514ca29a412d72e431293accf6b683f23c2bc3be7c5c63f28636d31d9"
Sep 14 13:40:08.562488 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:08.562454    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"]
Sep 14 13:40:09.288881 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:09.288851    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"]
Sep 14 13:40:23.580959 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:23.580915    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l" podUID="71102503-87cb-41cb-b452-61c418cd9f30" containerName="main" containerID="cri-o://963016c236e2adbecca6c2b6a4291ff99d892b5d3be132c5a4054c8493d5ed0b" gracePeriod=45
Sep 14 13:40:23.826218 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:23.826197    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:40:23.957725 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:23.957695    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24bm8\" (UniqueName: \"kubernetes.io/projected/71102503-87cb-41cb-b452-61c418cd9f30-kube-api-access-24bm8\") pod \"71102503-87cb-41cb-b452-61c418cd9f30\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") "
Sep 14 13:40:23.957910 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:23.957799    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-dshm\") pod \"71102503-87cb-41cb-b452-61c418cd9f30\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") "
Sep 14 13:40:23.957910 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:23.957844    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-model-cache\") pod \"71102503-87cb-41cb-b452-61c418cd9f30\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") "
Sep 14 13:40:23.957910 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:23.957869    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-home\") pod \"71102503-87cb-41cb-b452-61c418cd9f30\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") "
Sep 14 13:40:23.957910 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:23.957904    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71102503-87cb-41cb-b452-61c418cd9f30-tls-certs\") pod \"71102503-87cb-41cb-b452-61c418cd9f30\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") "
Sep 14 13:40:23.958131 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:23.957947    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-tmp-dir\") pod \"71102503-87cb-41cb-b452-61c418cd9f30\" (UID: \"71102503-87cb-41cb-b452-61c418cd9f30\") "
Sep 14 13:40:23.958131 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:23.958096    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-model-cache" (OuterVolumeSpecName: "model-cache") pod "71102503-87cb-41cb-b452-61c418cd9f30" (UID: "71102503-87cb-41cb-b452-61c418cd9f30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:40:23.958227 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:23.958152    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-home" (OuterVolumeSpecName: "home") pod "71102503-87cb-41cb-b452-61c418cd9f30" (UID: "71102503-87cb-41cb-b452-61c418cd9f30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:40:23.958267 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:23.958236    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "71102503-87cb-41cb-b452-61c418cd9f30" (UID: "71102503-87cb-41cb-b452-61c418cd9f30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:40:23.958317 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:23.958300    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-model-cache\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:40:23.958317 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:23.958313    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-home\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:40:23.960080 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:23.960057    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71102503-87cb-41cb-b452-61c418cd9f30-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "71102503-87cb-41cb-b452-61c418cd9f30" (UID: "71102503-87cb-41cb-b452-61c418cd9f30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:40:23.960343 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:23.960323    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-dshm" (OuterVolumeSpecName: "dshm") pod "71102503-87cb-41cb-b452-61c418cd9f30" (UID: "71102503-87cb-41cb-b452-61c418cd9f30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:40:23.960406 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:23.960348    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71102503-87cb-41cb-b452-61c418cd9f30-kube-api-access-24bm8" (OuterVolumeSpecName: "kube-api-access-24bm8") pod "71102503-87cb-41cb-b452-61c418cd9f30" (UID: "71102503-87cb-41cb-b452-61c418cd9f30"). InnerVolumeSpecName "kube-api-access-24bm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:40:24.060028 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.060000    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-dshm\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:40:24.060200 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.060044    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71102503-87cb-41cb-b452-61c418cd9f30-tls-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:40:24.060200 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.060109    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71102503-87cb-41cb-b452-61c418cd9f30-tmp-dir\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:40:24.060200 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.060122    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24bm8\" (UniqueName: \"kubernetes.io/projected/71102503-87cb-41cb-b452-61c418cd9f30-kube-api-access-24bm8\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:40:24.309253 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.309156    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8" podUID="7c13521d-e180-484e-b731-2adacb12c86c" containerName="main" containerID="cri-o://926744f1ccd6c99287cee78e65a050d2e7833470fb073b30894628cbfc885bbb" gracePeriod=45
Sep 14 13:40:24.429436 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.429404    2557 generic.go:356] "Generic (PLEG): container finished" podID="7c13521d-e180-484e-b731-2adacb12c86c" containerID="926744f1ccd6c99287cee78e65a050d2e7833470fb073b30894628cbfc885bbb" exitCode=0
Sep 14 13:40:24.429556 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.429473    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8" event={"ID":"7c13521d-e180-484e-b731-2adacb12c86c","Type":"ContainerDied","Data":"926744f1ccd6c99287cee78e65a050d2e7833470fb073b30894628cbfc885bbb"}
Sep 14 13:40:24.431104 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.431072    2557 generic.go:356] "Generic (PLEG): container finished" podID="71102503-87cb-41cb-b452-61c418cd9f30" containerID="963016c236e2adbecca6c2b6a4291ff99d892b5d3be132c5a4054c8493d5ed0b" exitCode=0
Sep 14 13:40:24.431244 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.431148    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"
Sep 14 13:40:24.431244 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.431181    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l" event={"ID":"71102503-87cb-41cb-b452-61c418cd9f30","Type":"ContainerDied","Data":"963016c236e2adbecca6c2b6a4291ff99d892b5d3be132c5a4054c8493d5ed0b"}
Sep 14 13:40:24.431244 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.431218    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l" event={"ID":"71102503-87cb-41cb-b452-61c418cd9f30","Type":"ContainerDied","Data":"0c4c65807228e5bea24ac06ba89fa3aefebe86b1c11c620c6715830f3f727449"}
Sep 14 13:40:24.431244 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.431241    2557 scope.go:117] "RemoveContainer" containerID="963016c236e2adbecca6c2b6a4291ff99d892b5d3be132c5a4054c8493d5ed0b"
Sep 14 13:40:24.441543 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.441517    2557 scope.go:117] "RemoveContainer" containerID="963016c236e2adbecca6c2b6a4291ff99d892b5d3be132c5a4054c8493d5ed0b"
Sep 14 13:40:24.441923 ip-10-0-129-61 kubenswrapper[2557]: E0914 13:40:24.441899    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963016c236e2adbecca6c2b6a4291ff99d892b5d3be132c5a4054c8493d5ed0b\": container with ID starting with 963016c236e2adbecca6c2b6a4291ff99d892b5d3be132c5a4054c8493d5ed0b not found: ID does not exist" containerID="963016c236e2adbecca6c2b6a4291ff99d892b5d3be132c5a4054c8493d5ed0b"
Sep 14 13:40:24.442025 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.441935    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963016c236e2adbecca6c2b6a4291ff99d892b5d3be132c5a4054c8493d5ed0b"} err="failed to get container status \"963016c236e2adbecca6c2b6a4291ff99d892b5d3be132c5a4054c8493d5ed0b\": rpc error: code = NotFound desc = could not find container \"963016c236e2adbecca6c2b6a4291ff99d892b5d3be132c5a4054c8493d5ed0b\": container with ID starting with 963016c236e2adbecca6c2b6a4291ff99d892b5d3be132c5a4054c8493d5ed0b not found: ID does not exist"
Sep 14 13:40:24.448989 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.448962    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"]
Sep 14 13:40:24.455143 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.455118    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-ff422ea2-kserve-676f858cff-d927l"]
Sep 14 13:40:24.568708 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.568683    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:40:24.664871 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.664848    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq4m5\" (UniqueName: \"kubernetes.io/projected/7c13521d-e180-484e-b731-2adacb12c86c-kube-api-access-dq4m5\") pod \"7c13521d-e180-484e-b731-2adacb12c86c\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") "
Sep 14 13:40:24.665236 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.664899    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c13521d-e180-484e-b731-2adacb12c86c-tls-certs\") pod \"7c13521d-e180-484e-b731-2adacb12c86c\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") "
Sep 14 13:40:24.665236 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.664942    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-home\") pod \"7c13521d-e180-484e-b731-2adacb12c86c\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") "
Sep 14 13:40:24.665236 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.664976    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-model-cache\") pod \"7c13521d-e180-484e-b731-2adacb12c86c\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") "
Sep 14 13:40:24.665236 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.665002    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-dshm\") pod \"7c13521d-e180-484e-b731-2adacb12c86c\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") "
Sep 14 13:40:24.665236 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.665034    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-tmp-dir\") pod \"7c13521d-e180-484e-b731-2adacb12c86c\" (UID: \"7c13521d-e180-484e-b731-2adacb12c86c\") "
Sep 14 13:40:24.665519 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.665230    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c13521d-e180-484e-b731-2adacb12c86c" (UID: "7c13521d-e180-484e-b731-2adacb12c86c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:40:24.665519 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.665236    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-home" (OuterVolumeSpecName: "home") pod "7c13521d-e180-484e-b731-2adacb12c86c" (UID: "7c13521d-e180-484e-b731-2adacb12c86c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:40:24.665519 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.665362    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-home\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:40:24.665519 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.665383    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-model-cache\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:40:24.665519 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.665442    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c13521d-e180-484e-b731-2adacb12c86c" (UID: "7c13521d-e180-484e-b731-2adacb12c86c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:40:24.666967 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.666935    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c13521d-e180-484e-b731-2adacb12c86c-kube-api-access-dq4m5" (OuterVolumeSpecName: "kube-api-access-dq4m5") pod "7c13521d-e180-484e-b731-2adacb12c86c" (UID: "7c13521d-e180-484e-b731-2adacb12c86c"). InnerVolumeSpecName "kube-api-access-dq4m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 13:40:24.667072 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.667022    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-dshm" (OuterVolumeSpecName: "dshm") pod "7c13521d-e180-484e-b731-2adacb12c86c" (UID: "7c13521d-e180-484e-b731-2adacb12c86c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 13:40:24.667072 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.667048    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c13521d-e180-484e-b731-2adacb12c86c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c13521d-e180-484e-b731-2adacb12c86c" (UID: "7c13521d-e180-484e-b731-2adacb12c86c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 13:40:24.766235 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.766215    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c13521d-e180-484e-b731-2adacb12c86c-tls-certs\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:40:24.766235 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.766233    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-dshm\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:40:24.766373 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.766241    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c13521d-e180-484e-b731-2adacb12c86c-tmp-dir\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:40:24.766373 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:24.766251    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq4m5\" (UniqueName: \"kubernetes.io/projected/7c13521d-e180-484e-b731-2adacb12c86c-kube-api-access-dq4m5\") on node \"ip-10-0-129-61.ec2.internal\" DevicePath \"\""
Sep 14 13:40:25.437625 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:25.437593    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"
Sep 14 13:40:25.437850 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:25.437628    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8" event={"ID":"7c13521d-e180-484e-b731-2adacb12c86c","Type":"ContainerDied","Data":"ba35c3de8d35dce5d2df6952b626134887eea73cd141cdf4702b5be3b26c4694"}
Sep 14 13:40:25.437850 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:25.437679    2557 scope.go:117] "RemoveContainer" containerID="926744f1ccd6c99287cee78e65a050d2e7833470fb073b30894628cbfc885bbb"
Sep 14 13:40:25.458276 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:25.458254    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"]
Sep 14 13:40:25.464174 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:25.464150    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-ff422ea2-kserve-6746d44d7-lcld8"]
Sep 14 13:40:26.048618 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:26.048587    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71102503-87cb-41cb-b452-61c418cd9f30" path="/var/lib/kubelet/pods/71102503-87cb-41cb-b452-61c418cd9f30/volumes"
Sep 14 13:40:26.048998 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:26.048968    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c13521d-e180-484e-b731-2adacb12c86c" path="/var/lib/kubelet/pods/7c13521d-e180-484e-b731-2adacb12c86c/volumes"
Sep 14 13:40:52.679692 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:52.679652    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f48dbdc44-94trb_769edf7b-d161-4eab-bca1-8a085d0c5357/authorino/0.log"
Sep 14 13:40:57.910435 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:57.910396    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gmncm_6fd5deac-72f2-4b25-9db8-82735b3ad959/manager/1.log"
Sep 14 13:40:58.023662 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:58.023637    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-fpq4w_120fe86c-4cf8-488d-bdb3-0c5a120d35dd/manager/0.log"
Sep 14 13:40:58.151032 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:58.151008    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c854ff6c6-glrk4_16d650af-bc70-4253-8e90-c270a7911fcc/manager/0.log"
Sep 14 13:40:58.269556 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:58.269533    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-xfqw6_a82fb799-541d-4bbc-9650-c22301c0bb25/server/0.log"
Sep 14 13:40:58.643440 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:58.643353    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-697cf97c89-j6tj8_ca27e799-89fb-4e04-9882-68c2fba75176/manager/0.log"
Sep 14 13:40:58.653817 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:58.653795    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-697cf97c89-j6tj8_ca27e799-89fb-4e04-9882-68c2fba75176/copy-manifests/0.log"
Sep 14 13:40:59.445715 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:59.445688    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29823210-qskrr_175254a6-1f95-48d7-a5b2-2cde0508afa8/cleanup/0.log"
Sep 14 13:40:59.557701 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:40:59.557680    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-jnsb2_b9e50a66-5a45-46e4-abda-28c7d3e1275c/postgres/0.log"
Sep 14 13:41:00.295755 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:00.295719    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f48dbdc44-94trb_769edf7b-d161-4eab-bca1-8a085d0c5357/authorino/0.log"
Sep 14 13:41:00.406684 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:00.406651    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-9q2vl_49fac7d8-720a-4482-a3bb-fec0ac32d1a4/manager/0.log"
Sep 14 13:41:00.520969 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:00.520942    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-4874m_30ca4937-6ff6-411a-8144-1d6c6733269d/manager/0.log"
Sep 14 13:41:00.627892 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:00.627819    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-kv2b4_b8271e98-9947-44ab-92b9-278948b9b3ec/kuadrant-console-plugin/0.log"
Sep 14 13:41:00.977559 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:00.977521    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-xgkvh_27b02f0d-9f27-4834-b312-ddc117f4433b/manager/0.log"
Sep 14 13:41:01.560498 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:01.560468    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f6f8d7966-fnc7x_1729b2a5-0803-4319-97ae-00eb46f0016c/kube-auth-proxy/0.log"
Sep 14 13:41:01.777318 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:01.777271    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2ht7f_1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60/istio-proxy/1.log"
Sep 14 13:41:01.892571 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:01.892499    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-q49w6_ab82241c-535a-426f-a3f3-7b902c041431/payload-pre-processing/0.log"
Sep 14 13:41:02.001502 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:02.001477    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zk2hb_692bcba1-e58a-4309-a7aa-4d13e276051b/payload-processing/0.log"
Sep 14 13:41:02.447719 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:02.447688    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-shdcx_8d80587c-6ef8-42ba-861f-b05f55b7938b/main/0.log"
Sep 14 13:41:02.845530 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:02.845456    2557 scope.go:117] "RemoveContainer" containerID="ce0a2bd7bdbcf26c88bde985a980f5f7b42bc6dd6a51fa1c3f388c183ff6922d"
Sep 14 13:41:02.853236 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:02.853218    2557 scope.go:117] "RemoveContainer" containerID="9a54700428603e7690988c64c7a892e9f5100b2389410e5fe5d79614e944d3be"
Sep 14 13:41:02.883189 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:02.883172    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jl27_85cf654b-351f-4180-a3da-a9f55f426bed/main/0.log"
Sep 14 13:41:08.521876 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:08.521845    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2dnwm/must-gather-w6xps"]
Sep 14 13:41:08.522351 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:08.522331    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c13521d-e180-484e-b731-2adacb12c86c" containerName="main"
Sep 14 13:41:08.522433 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:08.522355    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c13521d-e180-484e-b731-2adacb12c86c" containerName="main"
Sep 14 13:41:08.522433 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:08.522373    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71102503-87cb-41cb-b452-61c418cd9f30" containerName="main"
Sep 14 13:41:08.522433 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:08.522381    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="71102503-87cb-41cb-b452-61c418cd9f30" containerName="main"
Sep 14 13:41:08.522591 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:08.522480    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c13521d-e180-484e-b731-2adacb12c86c" containerName="main"
Sep 14 13:41:08.522591 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:08.522496    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="71102503-87cb-41cb-b452-61c418cd9f30" containerName="main"
Sep 14 13:41:08.527233 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:08.527210    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2dnwm/must-gather-w6xps"
Sep 14 13:41:08.530360 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:08.530272    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2dnwm\"/\"openshift-service-ca.crt\""
Sep 14 13:41:08.530490 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:08.530383    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2dnwm\"/\"default-dockercfg-5mfhs\""
Sep 14 13:41:08.531478 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:08.531461    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2dnwm\"/\"kube-root-ca.crt\""
Sep 14 13:41:08.537132 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:08.537109    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2dnwm/must-gather-w6xps"]
Sep 14 13:41:08.590433 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:08.590414    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srvm4\" (UniqueName: \"kubernetes.io/projected/607110b9-34cc-498b-aac8-f5645a3e87fb-kube-api-access-srvm4\") pod \"must-gather-w6xps\" (UID: \"607110b9-34cc-498b-aac8-f5645a3e87fb\") " pod="openshift-must-gather-2dnwm/must-gather-w6xps"
Sep 14 13:41:08.590542 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:08.590446    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/607110b9-34cc-498b-aac8-f5645a3e87fb-must-gather-output\") pod \"must-gather-w6xps\" (UID: \"607110b9-34cc-498b-aac8-f5645a3e87fb\") " pod="openshift-must-gather-2dnwm/must-gather-w6xps"
Sep 14 13:41:08.691618 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:08.691592    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srvm4\" (UniqueName: \"kubernetes.io/projected/607110b9-34cc-498b-aac8-f5645a3e87fb-kube-api-access-srvm4\") pod \"must-gather-w6xps\" (UID: \"607110b9-34cc-498b-aac8-f5645a3e87fb\") " pod="openshift-must-gather-2dnwm/must-gather-w6xps"
Sep 14 13:41:08.691712 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:08.691626    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/607110b9-34cc-498b-aac8-f5645a3e87fb-must-gather-output\") pod \"must-gather-w6xps\" (UID: \"607110b9-34cc-498b-aac8-f5645a3e87fb\") " pod="openshift-must-gather-2dnwm/must-gather-w6xps"
Sep 14 13:41:08.691966 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:08.691948    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/607110b9-34cc-498b-aac8-f5645a3e87fb-must-gather-output\") pod \"must-gather-w6xps\" (UID: \"607110b9-34cc-498b-aac8-f5645a3e87fb\") " pod="openshift-must-gather-2dnwm/must-gather-w6xps"
Sep 14 13:41:08.699786 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:08.699765    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srvm4\" (UniqueName: \"kubernetes.io/projected/607110b9-34cc-498b-aac8-f5645a3e87fb-kube-api-access-srvm4\") pod \"must-gather-w6xps\" (UID: \"607110b9-34cc-498b-aac8-f5645a3e87fb\") " pod="openshift-must-gather-2dnwm/must-gather-w6xps"
Sep 14 13:41:08.837528 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:08.837478    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2dnwm/must-gather-w6xps"
Sep 14 13:41:09.103171 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:09.103151    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2dnwm/must-gather-w6xps"]
Sep 14 13:41:09.105196 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:41:09.105166    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod607110b9_34cc_498b_aac8_f5645a3e87fb.slice/crio-d89f45c7b025683be90ea15a782fec7ee75abc829b3f31bc8d145b064a7aad97 WatchSource:0}: Error finding container d89f45c7b025683be90ea15a782fec7ee75abc829b3f31bc8d145b064a7aad97: Status 404 returned error can't find the container with id d89f45c7b025683be90ea15a782fec7ee75abc829b3f31bc8d145b064a7aad97
Sep 14 13:41:09.107254 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:09.107237    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 13:41:09.661185 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:09.661132    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2dnwm/must-gather-w6xps" event={"ID":"607110b9-34cc-498b-aac8-f5645a3e87fb","Type":"ContainerStarted","Data":"d89f45c7b025683be90ea15a782fec7ee75abc829b3f31bc8d145b064a7aad97"}
Sep 14 13:41:10.669778 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:10.669725    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2dnwm/must-gather-w6xps" event={"ID":"607110b9-34cc-498b-aac8-f5645a3e87fb","Type":"ContainerStarted","Data":"a206ee65442fe3e47d495bb5ed2fd87dbc6e5a41aeb74594a4c582ea3f718352"}
Sep 14 13:41:10.670160 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:10.669790    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2dnwm/must-gather-w6xps" event={"ID":"607110b9-34cc-498b-aac8-f5645a3e87fb","Type":"ContainerStarted","Data":"4cd098e76d0a8f2295d04fef5441b68afed80ab7992843bfe54a2a65ef2e19e5"}
Sep 14 13:41:10.687122 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:10.687066    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2dnwm/must-gather-w6xps" podStartSLOduration=2.017148987 podStartE2EDuration="2.687047614s" totalImagesPullingTime="669.898627ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:41:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:41:10.685812515 +0000 UTC m=+2589.120488034" watchObservedRunningTime="2026-09-14 13:41:10.687047614 +0000 UTC m=+2589.121723133"
Sep 14 13:41:11.352581 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:11.352553    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zgj9g_7eb7a234-f263-457d-9f06-c895898890ac/global-pull-secret-syncer/0.log"
Sep 14 13:41:11.429284 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:11.429259    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b97dp_e8eda6eb-1097-4a02-b60a-45eb7ddcc87e/konnectivity-agent/0.log"
Sep 14 13:41:11.491883 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:11.491856    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-61.ec2.internal_f85c20039e12da5048d2ed5c7f69bdf5/haproxy/0.log"
Sep 14 13:41:16.129268 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:16.129136    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f48dbdc44-94trb_769edf7b-d161-4eab-bca1-8a085d0c5357/authorino/0.log"
Sep 14 13:41:16.158701 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:16.158642    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-9q2vl_49fac7d8-720a-4482-a3bb-fec0ac32d1a4/manager/0.log"
Sep 14 13:41:16.188875 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:16.188846    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-4874m_30ca4937-6ff6-411a-8144-1d6c6733269d/manager/0.log"
Sep 14 13:41:16.218005 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:16.217977    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-kv2b4_b8271e98-9947-44ab-92b9-278948b9b3ec/kuadrant-console-plugin/0.log"
Sep 14 13:41:16.351550 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:16.351513    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-xgkvh_27b02f0d-9f27-4834-b312-ddc117f4433b/manager/0.log"
Sep 14 13:41:18.178766 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:18.178728    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bcclt_84e3b9c9-763e-405c-b22f-e5c2b07156d3/node-exporter/0.log"
Sep 14 13:41:18.211621 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:18.211590    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bcclt_84e3b9c9-763e-405c-b22f-e5c2b07156d3/kube-rbac-proxy/0.log"
Sep 14 13:41:18.242201 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:18.242165    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bcclt_84e3b9c9-763e-405c-b22f-e5c2b07156d3/init-textfile/0.log"
Sep 14 13:41:21.442765 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:21.442710    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h"]
Sep 14 13:41:21.448179 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:21.448155    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h"
Sep 14 13:41:21.459916 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:21.459561    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h"]
Sep 14 13:41:21.518303 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:21.518268    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f8f0199b-3e51-4b97-b408-b25972ac2199-proc\") pod \"perf-node-gather-daemonset-klj5h\" (UID: \"f8f0199b-3e51-4b97-b408-b25972ac2199\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h"
Sep 14 13:41:21.518490 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:21.518430    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f8f0199b-3e51-4b97-b408-b25972ac2199-lib-modules\") pod \"perf-node-gather-daemonset-klj5h\" (UID: \"f8f0199b-3e51-4b97-b408-b25972ac2199\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h"
Sep 14 13:41:21.518546 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:21.518507    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8f0199b-3e51-4b97-b408-b25972ac2199-sys\") pod \"perf-node-gather-daemonset-klj5h\" (UID: \"f8f0199b-3e51-4b97-b408-b25972ac2199\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h"
Sep 14 13:41:21.518546 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:21.518537    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f8f0199b-3e51-4b97-b408-b25972ac2199-podres\") pod \"perf-node-gather-daemonset-klj5h\" (UID: \"f8f0199b-3e51-4b97-b408-b25972ac2199\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h"
Sep 14 13:41:21.518656 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:21.518573    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2tgj\" (UniqueName: \"kubernetes.io/projected/f8f0199b-3e51-4b97-b408-b25972ac2199-kube-api-access-q2tgj\") pod \"perf-node-gather-daemonset-klj5h\" (UID: \"f8f0199b-3e51-4b97-b408-b25972ac2199\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h"
Sep 14 13:41:21.620068 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:21.620034    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8f0199b-3e51-4b97-b408-b25972ac2199-sys\") pod \"perf-node-gather-daemonset-klj5h\" (UID: \"f8f0199b-3e51-4b97-b408-b25972ac2199\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h"
Sep 14 13:41:21.620328 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:21.620299    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8f0199b-3e51-4b97-b408-b25972ac2199-sys\") pod \"perf-node-gather-daemonset-klj5h\" (UID: \"f8f0199b-3e51-4b97-b408-b25972ac2199\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h"
Sep 14 13:41:21.620427 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:21.620307    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f8f0199b-3e51-4b97-b408-b25972ac2199-podres\") pod \"perf-node-gather-daemonset-klj5h\" (UID: \"f8f0199b-3e51-4b97-b408-b25972ac2199\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h"
Sep 14 13:41:21.620427 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:21.620391    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2tgj\" (UniqueName: \"kubernetes.io/projected/f8f0199b-3e51-4b97-b408-b25972ac2199-kube-api-access-q2tgj\") pod \"perf-node-gather-daemonset-klj5h\" (UID: \"f8f0199b-3e51-4b97-b408-b25972ac2199\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h"
Sep 14 13:41:21.620550 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:21.620463    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f8f0199b-3e51-4b97-b408-b25972ac2199-proc\") pod \"perf-node-gather-daemonset-klj5h\" (UID: \"f8f0199b-3e51-4b97-b408-b25972ac2199\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h"
Sep 14 13:41:21.620610 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:21.620569    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f8f0199b-3e51-4b97-b408-b25972ac2199-lib-modules\") pod \"perf-node-gather-daemonset-klj5h\" (UID: \"f8f0199b-3e51-4b97-b408-b25972ac2199\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h"
Sep 14 13:41:21.620796 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:21.620772    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f8f0199b-3e51-4b97-b408-b25972ac2199-lib-modules\") pod \"perf-node-gather-daemonset-klj5h\" (UID: \"f8f0199b-3e51-4b97-b408-b25972ac2199\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h"
Sep 14 13:41:21.620890 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:21.620842    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f8f0199b-3e51-4b97-b408-b25972ac2199-proc\") pod \"perf-node-gather-daemonset-klj5h\" (UID: \"f8f0199b-3e51-4b97-b408-b25972ac2199\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h"
Sep 14 13:41:21.621084 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:21.621064    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f8f0199b-3e51-4b97-b408-b25972ac2199-podres\") pod \"perf-node-gather-daemonset-klj5h\" (UID: \"f8f0199b-3e51-4b97-b408-b25972ac2199\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h"
Sep 14 13:41:21.631266 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:21.631243    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2tgj\" (UniqueName: \"kubernetes.io/projected/f8f0199b-3e51-4b97-b408-b25972ac2199-kube-api-access-q2tgj\") pod \"perf-node-gather-daemonset-klj5h\" (UID: \"f8f0199b-3e51-4b97-b408-b25972ac2199\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h"
Sep 14 13:41:21.766882 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:21.766803    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h"
Sep 14 13:41:22.052816 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:22.052793    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h"]
Sep 14 13:41:22.056085 ip-10-0-129-61 kubenswrapper[2557]: W0914 13:41:22.056057    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf8f0199b_3e51_4b97_b408_b25972ac2199.slice/crio-784d8d9a6c0907790a2fe0c5e07b3a75b6b4225fd6e33170f49f1a316781534d WatchSource:0}: Error finding container 784d8d9a6c0907790a2fe0c5e07b3a75b6b4225fd6e33170f49f1a316781534d: Status 404 returned error can't find the container with id 784d8d9a6c0907790a2fe0c5e07b3a75b6b4225fd6e33170f49f1a316781534d
Sep 14 13:41:22.756395 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:22.756348    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h" event={"ID":"f8f0199b-3e51-4b97-b408-b25972ac2199","Type":"ContainerStarted","Data":"faadd6331f466788a55d6f104c064d6ea13a268ec2ad4dbf4a2b1a36e407455c"}
Sep 14 13:41:22.757024 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:22.756467    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h" event={"ID":"f8f0199b-3e51-4b97-b408-b25972ac2199","Type":"ContainerStarted","Data":"784d8d9a6c0907790a2fe0c5e07b3a75b6b4225fd6e33170f49f1a316781534d"}
Sep 14 13:41:22.757267 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:22.757239    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h"
Sep 14 13:41:22.774402 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:22.774342    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h" podStartSLOduration=1.774323927 podStartE2EDuration="1.774323927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 13:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:41:22.77343588 +0000 UTC m=+2601.208111400" watchObservedRunningTime="2026-09-14 13:41:22.774323927 +0000 UTC m=+2601.208999447"
Sep 14 13:41:23.381526 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:23.381491    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gw4vc_f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6/dns/0.log"
Sep 14 13:41:23.402512 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:23.402484    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gw4vc_f3d17f3f-bf92-4d1f-a6b8-b1efdd0781e6/kube-rbac-proxy/0.log"
Sep 14 13:41:23.492242 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:23.492213    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nhvd2_e41b38b6-a3f2-4093-977d-c44ada18fdcb/dns-node-resolver/0.log"
Sep 14 13:41:24.113045 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:24.113017    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5vcr5_b4458034-12ab-40d0-93c8-77de00c6ffb7/node-ca/0.log"
Sep 14 13:41:25.891055 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:25.890984    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f6f8d7966-fnc7x_1729b2a5-0803-4319-97ae-00eb46f0016c/kube-auth-proxy/0.log"
Sep 14 13:41:25.939213 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:25.939188    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2ht7f_1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60/istio-proxy/0.log"
Sep 14 13:41:25.942537 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:25.942515    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2ht7f_1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60/istio-proxy/1.log"
Sep 14 13:41:25.978236 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:25.978212    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-q49w6_ab82241c-535a-426f-a3f3-7b902c041431/payload-pre-processing/0.log"
Sep 14 13:41:26.009490 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:26.009460    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zk2hb_692bcba1-e58a-4309-a7aa-4d13e276051b/payload-processing/0.log"
Sep 14 13:41:26.911933 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:26.911901    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jmcw8_d44e13a7-7e76-4eee-873f-68bf4066ac29/serve-healthcheck-canary/0.log"
Sep 14 13:41:27.478341 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:27.478316    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-97j4b_5825fb8c-8360-46b3-9f34-57ab3a5a3e4e/kube-rbac-proxy/0.log"
Sep 14 13:41:27.496508 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:27.496480    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-97j4b_5825fb8c-8360-46b3-9f34-57ab3a5a3e4e/exporter/0.log"
Sep 14 13:41:27.517409 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:27.517385    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-97j4b_5825fb8c-8360-46b3-9f34-57ab3a5a3e4e/extractor/0.log"
Sep 14 13:41:29.625068 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:29.625037    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gmncm_6fd5deac-72f2-4b25-9db8-82735b3ad959/manager/1.log"
Sep 14 13:41:29.625452 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:29.625180    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gmncm_6fd5deac-72f2-4b25-9db8-82735b3ad959/manager/0.log"
Sep 14 13:41:29.780110 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:29.780085    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-klj5h"
Sep 14 13:41:29.946867 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:29.946832    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-fpq4w_120fe86c-4cf8-488d-bdb3-0c5a120d35dd/manager/0.log"
Sep 14 13:41:30.003287 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:30.003251    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c854ff6c6-glrk4_16d650af-bc70-4253-8e90-c270a7911fcc/manager/0.log"
Sep 14 13:41:30.022190 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:30.022166    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-xfqw6_a82fb799-541d-4bbc-9650-c22301c0bb25/server/0.log"
Sep 14 13:41:30.300705 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:30.300627    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-697cf97c89-j6tj8_ca27e799-89fb-4e04-9882-68c2fba75176/manager/0.log"
Sep 14 13:41:30.315748 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:30.315708    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-697cf97c89-j6tj8_ca27e799-89fb-4e04-9882-68c2fba75176/copy-manifests/0.log"
Sep 14 13:41:31.719386 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:31.719359    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-jbkrl_dc1fed35-c3ce-4962-aadc-1529061210d6/openshift-lws-operator/0.log"
Sep 14 13:41:31.955703 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:31.955673    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f6f8d7966-fnc7x_1729b2a5-0803-4319-97ae-00eb46f0016c/kube-auth-proxy/0.log"
Sep 14 13:41:32.005846 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:32.005769    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2ht7f_1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60/istio-proxy/0.log"
Sep 14 13:41:32.008922 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:32.008894    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2ht7f_1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60/istio-proxy/1.log"
Sep 14 13:41:32.036808 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:32.036776    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-q49w6_ab82241c-535a-426f-a3f3-7b902c041431/payload-pre-processing/0.log"
Sep 14 13:41:32.064270 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:32.064218    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zk2hb_692bcba1-e58a-4309-a7aa-4d13e276051b/payload-processing/0.log"
Sep 14 13:41:34.519375 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:34.519337    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f6f8d7966-fnc7x_1729b2a5-0803-4319-97ae-00eb46f0016c/kube-auth-proxy/0.log"
Sep 14 13:41:34.556880 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:34.556851    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2ht7f_1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60/istio-proxy/0.log"
Sep 14 13:41:34.559571 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:34.559546    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2ht7f_1fe5fabe-2ec0-4dae-9e21-4fc8639bbc60/istio-proxy/1.log"
Sep 14 13:41:34.579329 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:34.579305    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-q49w6_ab82241c-535a-426f-a3f3-7b902c041431/payload-pre-processing/0.log"
Sep 14 13:41:34.600898 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:34.600878    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zk2hb_692bcba1-e58a-4309-a7aa-4d13e276051b/payload-processing/0.log"
Sep 14 13:41:38.228424 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:38.228390    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zcv75_f26cce38-f34e-4f48-9a02-88e6674fe266/kube-multus-additional-cni-plugins/0.log"
Sep 14 13:41:38.263432 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:38.263407    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zcv75_f26cce38-f34e-4f48-9a02-88e6674fe266/egress-router-binary-copy/0.log"
Sep 14 13:41:38.291017 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:38.290997    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zcv75_f26cce38-f34e-4f48-9a02-88e6674fe266/cni-plugins/0.log"
Sep 14 13:41:38.324125 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:38.324106    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zcv75_f26cce38-f34e-4f48-9a02-88e6674fe266/bond-cni-plugin/0.log"
Sep 14 13:41:38.357119 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:38.357100    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zcv75_f26cce38-f34e-4f48-9a02-88e6674fe266/routeoverride-cni/0.log"
Sep 14 13:41:38.389085 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:38.389066    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zcv75_f26cce38-f34e-4f48-9a02-88e6674fe266/whereabouts-cni-bincopy/0.log"
Sep 14 13:41:38.425035 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:38.425016    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zcv75_f26cce38-f34e-4f48-9a02-88e6674fe266/whereabouts-cni/0.log"
Sep 14 13:41:38.646953 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:38.646884    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cz95j_3416c159-66b8-4cc1-844a-856cf03e8bc8/kube-multus/0.log"
Sep 14 13:41:38.897970 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:38.897892    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nl2vh_898a4541-4e3f-4276-b037-ff179ac5fc52/network-metrics-daemon/0.log"
Sep 14 13:41:38.928259 ip-10-0-129-61 kubenswrapper[2557]: I0914 13:41:38.928233    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nl2vh_898a4541-4e3f-4276-b037-ff179ac5fc52/kube-rbac-proxy/0.log"