Sep 15 07:18:15.354547 ip-10-0-129-76 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 07:18:15.693555 ip-10-0-129-76 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 07:18:15.693555 ip-10-0-129-76 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 07:18:15.693555 ip-10-0-129-76 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 07:18:15.693555 ip-10-0-129-76 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 07:18:15.693555 ip-10-0-129-76 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 07:18:15.694358 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.694274    2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 07:18:15.697748 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697732    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 07:18:15.697748 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697748    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 07:18:15.697812 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697754    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 07:18:15.697812 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697758    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 07:18:15.697812 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697762    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 07:18:15.697812 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697765    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 07:18:15.697812 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697768    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 07:18:15.697812 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697770    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 07:18:15.697812 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697773    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 07:18:15.697812 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697776    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 07:18:15.697812 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697779    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 07:18:15.697812 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697781    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 07:18:15.697812 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697784    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 07:18:15.697812 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697787    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 07:18:15.697812 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697790    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 07:18:15.697812 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697793    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 07:18:15.697812 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697796    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 07:18:15.697812 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697798    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 07:18:15.697812 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697801    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 07:18:15.697812 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697803    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 07:18:15.697812 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697806    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 07:18:15.697812 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697808    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 07:18:15.698309 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697811    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 07:18:15.698309 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697813    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 07:18:15.698309 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697816    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 07:18:15.698309 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697819    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 07:18:15.698309 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697822    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 07:18:15.698309 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697824    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 07:18:15.698309 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697827    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 07:18:15.698309 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697830    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 07:18:15.698309 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697833    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 07:18:15.698309 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697835    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 07:18:15.698309 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697838    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 07:18:15.698309 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697841    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 07:18:15.698309 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697843    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 07:18:15.698309 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697846    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 07:18:15.698309 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697848    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 07:18:15.698309 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697851    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 07:18:15.698309 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697853    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 07:18:15.698309 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697855    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 07:18:15.698309 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697858    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 07:18:15.698772 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697861    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 07:18:15.698772 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697863    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 07:18:15.698772 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697866    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 07:18:15.698772 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697868    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 07:18:15.698772 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697870    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 07:18:15.698772 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697873    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 07:18:15.698772 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697876    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 07:18:15.698772 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697880    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 07:18:15.698772 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697884    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 07:18:15.698772 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697886    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 07:18:15.698772 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697889    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 07:18:15.698772 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697892    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 07:18:15.698772 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697902    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 07:18:15.698772 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697906    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 07:18:15.698772 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697916    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 07:18:15.698772 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697920    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 07:18:15.698772 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697923    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 07:18:15.698772 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697926    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 07:18:15.698772 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697929    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 07:18:15.698772 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697932    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 07:18:15.699270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697935    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 07:18:15.699270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697938    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 07:18:15.699270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697940    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 07:18:15.699270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697943    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 07:18:15.699270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697946    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 07:18:15.699270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697948    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 07:18:15.699270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697950    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 07:18:15.699270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697953    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 07:18:15.699270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697956    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 07:18:15.699270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697958    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 07:18:15.699270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697961    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 07:18:15.699270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697964    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 07:18:15.699270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697966    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 07:18:15.699270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697969    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 07:18:15.699270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697971    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 07:18:15.699270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697974    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 07:18:15.699270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697976    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 07:18:15.699270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697980    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 07:18:15.699270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697983    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 07:18:15.699270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697990    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 07:18:15.699769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697992    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 07:18:15.699769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697995    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 07:18:15.699769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.697998    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 07:18:15.699769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698000    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 07:18:15.699769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698003    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 07:18:15.699769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698006    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 07:18:15.699769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698009    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 07:18:15.699769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698012    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 07:18:15.699769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698014    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 07:18:15.699769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698017    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 07:18:15.699769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698020    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 07:18:15.699769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698023    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 07:18:15.699769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698026    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 07:18:15.699769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698028    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 07:18:15.699769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698031    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 07:18:15.699769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698033    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 07:18:15.699769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698035    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 07:18:15.699769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698038    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 07:18:15.699769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698769    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 07:18:15.699769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698777    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 07:18:15.700269 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698780    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 07:18:15.700269 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698783    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 07:18:15.700269 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698787    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 07:18:15.700269 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698790    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 07:18:15.700269 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698793    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 07:18:15.700269 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698795    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 07:18:15.700269 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698798    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 07:18:15.700269 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698801    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 07:18:15.700269 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698803    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 07:18:15.700269 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698805    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 07:18:15.700269 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698808    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 07:18:15.700269 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698810    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 07:18:15.700269 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698813    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 07:18:15.700269 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698816    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 07:18:15.700269 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698819    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 07:18:15.700269 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698821    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 07:18:15.700269 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698825    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 07:18:15.700269 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698829    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 07:18:15.700269 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698832    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 07:18:15.700269 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698835    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 07:18:15.700769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698838    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 07:18:15.700769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698840    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 07:18:15.700769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698843    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 07:18:15.700769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698846    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 07:18:15.700769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698848    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 07:18:15.700769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698851    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 07:18:15.700769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698853    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 07:18:15.700769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698856    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 07:18:15.700769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698859    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 07:18:15.700769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698861    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 07:18:15.700769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698864    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 07:18:15.700769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698866    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 07:18:15.700769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698869    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 07:18:15.700769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698871    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 07:18:15.700769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698874    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 07:18:15.700769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698876    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 07:18:15.700769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698879    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 07:18:15.700769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698881    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 07:18:15.700769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698884    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 07:18:15.700769 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698889    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 07:18:15.701270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698892    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 07:18:15.701270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698895    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 07:18:15.701270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698898    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 07:18:15.701270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698901    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 07:18:15.701270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698904    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 07:18:15.701270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698906    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 07:18:15.701270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698909    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 07:18:15.701270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698911    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 07:18:15.701270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698914    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 07:18:15.701270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698917    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 07:18:15.701270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698920    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 07:18:15.701270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698923    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 07:18:15.701270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698925    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 07:18:15.701270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698928    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 07:18:15.701270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698930    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 07:18:15.701270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698933    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 07:18:15.701270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698936    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 07:18:15.701270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698938    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 07:18:15.701270 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698940    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 07:18:15.701738 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698944    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 07:18:15.701738 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698948    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 07:18:15.701738 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698950    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 07:18:15.701738 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698953    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 07:18:15.701738 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698956    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 07:18:15.701738 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698958    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 07:18:15.701738 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698961    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 07:18:15.701738 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698963    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 07:18:15.701738 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698966    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 07:18:15.701738 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698968    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 07:18:15.701738 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698971    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 07:18:15.701738 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698973    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 07:18:15.701738 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698977    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 07:18:15.701738 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698979    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 07:18:15.701738 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698981    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 07:18:15.701738 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698984    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 07:18:15.701738 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698986    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 07:18:15.701738 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698989    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 07:18:15.701738 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698991    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 07:18:15.702211 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698994    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 07:18:15.702211 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.698997    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 07:18:15.702211 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699000    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 07:18:15.702211 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699002    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 07:18:15.702211 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699007    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 07:18:15.702211 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699010    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 07:18:15.702211 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699012    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 07:18:15.702211 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699014    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 07:18:15.702211 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699017    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 07:18:15.702211 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699019    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 07:18:15.702211 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699022    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 07:18:15.702211 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699024    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 07:18:15.702211 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699027    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 07:18:15.702211 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699029    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 07:18:15.702211 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699032    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 07:18:15.702211 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699034    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 07:18:15.702211 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699037    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 07:18:15.702211 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699039    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 07:18:15.702211 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699042    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 07:18:15.702681 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699425    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 07:18:15.702681 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699430    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 07:18:15.702681 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699433    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 07:18:15.702681 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699436    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 07:18:15.702681 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699439    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 07:18:15.702681 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699442    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 07:18:15.702681 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699444    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 07:18:15.702681 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699447    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 07:18:15.702681 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699450    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 07:18:15.702681 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699454    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 07:18:15.702681 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699458    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 07:18:15.702681 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699461    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 07:18:15.702681 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699464    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 07:18:15.702681 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699467    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 07:18:15.702681 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699470    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 07:18:15.702681 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699474    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 07:18:15.702681 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699477    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 07:18:15.702681 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699481    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 07:18:15.702681 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699484    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 07:18:15.703159 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699487    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 07:18:15.703159 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699490    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 07:18:15.703159 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699494    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 07:18:15.703159 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699497    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 07:18:15.703159 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699500    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 07:18:15.703159 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699503    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 07:18:15.703159 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699506    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 07:18:15.703159 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699508    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 07:18:15.703159 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699511    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 07:18:15.703159 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699513    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 07:18:15.703159 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699516    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 07:18:15.703159 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699519    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 07:18:15.703159 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699521    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 07:18:15.703159 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699523    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 07:18:15.703159 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699527    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 07:18:15.703159 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699530    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 07:18:15.703159 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699533    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 07:18:15.703159 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699535    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 07:18:15.703159 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699538    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 07:18:15.703632 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699540    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 07:18:15.703632 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699543    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 07:18:15.703632 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699545    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 07:18:15.703632 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699548    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 07:18:15.703632 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699550    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 07:18:15.703632 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699553    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 07:18:15.703632 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699555    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 07:18:15.703632 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699558    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 07:18:15.703632 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699561    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 07:18:15.703632 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699564    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 07:18:15.703632 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699566    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 07:18:15.703632 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699569    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 07:18:15.703632 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699572    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 07:18:15.703632 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699575    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 07:18:15.703632 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699577    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 07:18:15.703632 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699580    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 07:18:15.703632 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699582    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 07:18:15.703632 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699585    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 07:18:15.703632 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699587    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 07:18:15.703632 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699590    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 07:18:15.704126 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699592    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 07:18:15.704126 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699595    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 07:18:15.704126 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699598    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 07:18:15.704126 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699600    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 07:18:15.704126 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699603    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 07:18:15.704126 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699606    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 07:18:15.704126 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699608    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 07:18:15.704126 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699610    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 07:18:15.704126 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699613    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 07:18:15.704126 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699615    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 07:18:15.704126 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699621    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 07:18:15.704126 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699624    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 07:18:15.704126 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699626    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 07:18:15.704126 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699629    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 07:18:15.704126 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699632    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 07:18:15.704126 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699636    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 07:18:15.704126 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699638    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 07:18:15.704126 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699641    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 07:18:15.704126 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699644    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 07:18:15.704126 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699646    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 07:18:15.704656 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699648    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 07:18:15.704656 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699652    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 07:18:15.704656 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699654    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 07:18:15.704656 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699657    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 07:18:15.704656 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699660    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 07:18:15.704656 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699662    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 07:18:15.704656 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699665    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 07:18:15.704656 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699668    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 07:18:15.704656 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699670    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 07:18:15.704656 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699672    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 07:18:15.704656 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699675    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 07:18:15.704656 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699677    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 07:18:15.704656 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699680    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 07:18:15.704656 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699682    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 07:18:15.704656 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699685    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 07:18:15.704656 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699687    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 07:18:15.704656 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699690    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 07:18:15.704656 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699693    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 07:18:15.704656 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699696    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 07:18:15.704656 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699699    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.699701    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.699779    2571 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.699814    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.699821    2571 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.699826    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.699830    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.699833    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.699838    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.699842    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.699845    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.699848    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.699851    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.699855    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.699858    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.699861    2571 flags.go:64] FLAG: --cgroup-root=""
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.699865    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.699869    2571 flags.go:64] FLAG: --client-ca-file=""
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.699872    2571 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.699875    2571 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700448    2571 flags.go:64] FLAG: --cluster-domain=""
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700452    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700455    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700459    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 07:18:15.705152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700462    2571 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700466    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700469    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700472    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700475    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700478    2571 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700481    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700484    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700487    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700490    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700494    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700498    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700501    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700504    2571 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700507    2571 flags.go:64] FLAG: --enable-server="true"
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700510    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700519    2571 flags.go:64] FLAG: --event-burst="100"
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700522    2571 flags.go:64] FLAG: --event-qps="50"
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700525    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700529    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700532    2571 flags.go:64] FLAG: --eviction-hard=""
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700536    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700539    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700542    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700546    2571 flags.go:64] FLAG: --eviction-soft=""
Sep 15 07:18:15.705776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700549    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700551    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700555    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700558    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700561    2571 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700563    2571 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700566    2571 flags.go:64] FLAG: --feature-gates=""
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700570    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700573    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700576    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700580    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700583    2571 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700586    2571 flags.go:64] FLAG: --help="false"
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700589    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-129-76.ec2.internal"
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700592    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700595    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700598    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700602    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700605    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700608    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700611    2571 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700614    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700618    2571 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 07:18:15.706411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700621    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700625    2571 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700627    2571 flags.go:64] FLAG: --kube-reserved=""
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700630    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700634    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700637    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700639    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700642    2571 flags.go:64] FLAG: --lock-file=""
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700645    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700648    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700651    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700659    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700662    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700665    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700668    2571 flags.go:64] FLAG: --logging-format="text"
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700671    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700674    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700677    2571 flags.go:64] FLAG: --manifest-url=""
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700680    2571 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700684    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700688    2571 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700692    2571 flags.go:64] FLAG: --max-pods="110"
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700695    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700698    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700701    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 07:18:15.706999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700704    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700707    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700711    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700714    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700721    2571 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700724    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700727    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700730    2571 flags.go:64] FLAG: --pod-cidr=""
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700733    2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700739    2571 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700742    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700745    2571 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700748    2571 flags.go:64] FLAG: --port="10250"
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700751    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700754    2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c38f890321f7dd19"
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700757    2571 flags.go:64] FLAG: --qos-reserved=""
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700760    2571 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700763    2571 flags.go:64] FLAG: --register-node="true"
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700767    2571 flags.go:64] FLAG: --register-with-taints=""
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700770    2571 flags.go:64] FLAG: --registry-burst="10"
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700773    2571 flags.go:64] FLAG: --registry-qps="5"
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700776    2571 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700779    2571 flags.go:64] FLAG: --reserved-memory=""
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700783    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700786    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 07:18:15.707955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700789    2571 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700792    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700795    2571 flags.go:64] FLAG: --runonce="false"
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700798    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700801    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700804    2571 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700807    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700810    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700815    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700818    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700821    2571 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700824    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700827    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700830    2571 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700834    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700837    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700840    2571 flags.go:64] FLAG: --system-cgroups=""
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700843    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700848    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700852    2571 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700855    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700859    2571 flags.go:64] FLAG: --tls-min-version=""
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700862    2571 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700865    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700867    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 07:18:15.708698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700871    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 07:18:15.709351 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700874    2571 flags.go:64] FLAG: --v="2"
Sep 15 07:18:15.709351 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700878    2571 flags.go:64] FLAG: --version="false"
Sep 15 07:18:15.709351 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700882    2571 flags.go:64] FLAG: --vmodule=""
Sep 15 07:18:15.709351 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700885    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 07:18:15.709351 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.700889    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 07:18:15.709351 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701005    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 07:18:15.709351 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701009    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 07:18:15.709351 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701012    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 07:18:15.709351 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701015    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 07:18:15.709351 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701018    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 07:18:15.709351 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701020    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 07:18:15.709351 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701023    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 07:18:15.709351 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701025    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 07:18:15.709351 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701028    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 07:18:15.709351 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701032    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 07:18:15.709351 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701034    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 07:18:15.709351 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701037    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 07:18:15.709351 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701040    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 07:18:15.709351 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701042    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 07:18:15.709351 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701045    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 07:18:15.709351 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701052    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 07:18:15.709916 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701054    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 07:18:15.709916 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701057    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 07:18:15.709916 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701060    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 07:18:15.709916 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701063    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 07:18:15.709916 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701065    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 07:18:15.709916 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701068    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 07:18:15.709916 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701071    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 07:18:15.709916 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701074    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 07:18:15.709916 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701076    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 07:18:15.709916 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701079    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 07:18:15.709916 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701081    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 07:18:15.709916 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701084    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 07:18:15.709916 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701087    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 07:18:15.709916 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701090    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 07:18:15.709916 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701092    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 07:18:15.709916 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701095    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 07:18:15.709916 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701098    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 07:18:15.709916 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701100    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 07:18:15.709916 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701103    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 07:18:15.709916 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701105    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 07:18:15.710419 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701108    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 07:18:15.710419 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701111    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 07:18:15.710419 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701113    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 07:18:15.710419 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701116    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 07:18:15.710419 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701118    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 07:18:15.710419 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701122    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 07:18:15.710419 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701126    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 07:18:15.710419 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701130    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 07:18:15.710419 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701133    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 07:18:15.710419 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701135    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 07:18:15.710419 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701138    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 07:18:15.710419 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701142    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 07:18:15.710419 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701145    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 07:18:15.710419 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701147    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 07:18:15.710419 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701150    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 07:18:15.710419 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701153    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 07:18:15.710419 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701155    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 07:18:15.710419 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701158    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 07:18:15.710419 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701161    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 07:18:15.710878 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701163    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 07:18:15.710878 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701166    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 07:18:15.710878 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701169    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 07:18:15.710878 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701173    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 07:18:15.710878 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701177    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 07:18:15.710878 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701182    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 07:18:15.710878 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701185    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 07:18:15.710878 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701188    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 07:18:15.710878 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701191    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 07:18:15.710878 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701194    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 07:18:15.710878 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701196    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 07:18:15.710878 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701198    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 07:18:15.710878 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701201    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 07:18:15.710878 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701203    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 07:18:15.710878 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701206    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 07:18:15.710878 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701208    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 07:18:15.710878 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701211    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 07:18:15.710878 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701214    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 07:18:15.710878 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701217    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 07:18:15.711441 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701220    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 07:18:15.711441 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701222    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 07:18:15.711441 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701225    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 07:18:15.711441 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701228    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 07:18:15.711441 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701230    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 07:18:15.711441 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701235    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 07:18:15.711441 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701237    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 07:18:15.711441 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701240    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 07:18:15.711441 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701243    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 07:18:15.711441 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701245    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 07:18:15.711441 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701248    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 07:18:15.711441 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701262    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 07:18:15.711441 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701265    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 07:18:15.711441 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701269    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 07:18:15.711441 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701272    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 07:18:15.711441 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701275    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 07:18:15.711441 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701279    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 07:18:15.711441 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701282    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 07:18:15.711441 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701285    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 07:18:15.711441 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701288    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 07:18:15.711936 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701291    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 07:18:15.711936 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701294    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 07:18:15.711936 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701296    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 07:18:15.711936 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701299    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 07:18:15.711936 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.701302    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 07:18:15.711936 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.701865    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 07:18:15.711936 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.710186    2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 07:18:15.711936 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.710202    2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 07:18:15.711936 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.710216    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 07:18:15.711936 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.710221    2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 07:18:15.711936 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710296    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 07:18:15.711936 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710302    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 07:18:15.711936 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710305    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 07:18:15.711936 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710308    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 07:18:15.712312 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710311    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 07:18:15.712312 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710314    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 07:18:15.712312 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710317    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 07:18:15.712312 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710321    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 07:18:15.712312 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710326    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 07:18:15.712312 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710329    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 07:18:15.712312 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710331    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 07:18:15.712312 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710335    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 07:18:15.712312 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710339    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 07:18:15.712312 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710343    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 07:18:15.712312 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710346    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 07:18:15.712312 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710349    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 07:18:15.712312 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710352    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 07:18:15.712312 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710354    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 07:18:15.712312 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710357    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 07:18:15.712312 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710360    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 07:18:15.712312 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710362    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 07:18:15.712312 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710365    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 07:18:15.712312 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710367    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 07:18:15.712789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710371    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 07:18:15.712789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710373    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 07:18:15.712789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710376    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 07:18:15.712789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710378    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 07:18:15.712789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710381    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 07:18:15.712789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710384    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 07:18:15.712789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710388    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 07:18:15.712789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710391    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 07:18:15.712789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710394    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 07:18:15.712789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710396    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 07:18:15.712789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710399    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 07:18:15.712789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710401    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 07:18:15.712789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710404    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 07:18:15.712789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710407    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 07:18:15.712789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710410    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 07:18:15.712789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710412    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 07:18:15.712789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710415    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 07:18:15.712789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710417    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 07:18:15.712789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710420    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 07:18:15.712789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710423    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 07:18:15.713333 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710426    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 07:18:15.713333 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710428    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 07:18:15.713333 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710431    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 07:18:15.713333 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710435    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 07:18:15.713333 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710439    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 07:18:15.713333 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710442    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 07:18:15.713333 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710446    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 07:18:15.713333 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710448    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 07:18:15.713333 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710451    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 07:18:15.713333 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710454    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 07:18:15.713333 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710457    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 07:18:15.713333 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710460    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 07:18:15.713333 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710463    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 07:18:15.713333 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710466    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 07:18:15.713333 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710470    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 07:18:15.713333 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710473    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 07:18:15.713333 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710475    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 07:18:15.713333 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710478    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 07:18:15.713333 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710481    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 07:18:15.713789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710483    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 07:18:15.713789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710486    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 07:18:15.713789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710489    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 07:18:15.713789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710491    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 07:18:15.713789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710494    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 07:18:15.713789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710496    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 07:18:15.713789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710499    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 07:18:15.713789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710501    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 07:18:15.713789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710504    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 07:18:15.713789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710506    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 07:18:15.713789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710509    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 07:18:15.713789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710512    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 07:18:15.713789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710514    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 07:18:15.713789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710517    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 07:18:15.713789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710519    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 07:18:15.713789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710522    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 07:18:15.713789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710525    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 07:18:15.713789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710528    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 07:18:15.713789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710532    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 07:18:15.713789 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710534    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 07:18:15.714304 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710537    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 07:18:15.714304 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710539    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 07:18:15.714304 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710542    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 07:18:15.714304 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710545    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 07:18:15.714304 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710547    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 07:18:15.714304 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710550    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 07:18:15.714304 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710552    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 07:18:15.714304 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710555    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 07:18:15.714304 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710558    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 07:18:15.714304 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710560    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 07:18:15.714304 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710563    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 07:18:15.714304 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710566    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 07:18:15.714304 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710569    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 07:18:15.714304 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710571    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 07:18:15.714304 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710574    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 07:18:15.714304 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710576    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 07:18:15.714304 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710579    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 07:18:15.714768 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.710584    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 07:18:15.714768 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710692    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 07:18:15.714768 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710696    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 07:18:15.714768 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710700    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 07:18:15.714768 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710703    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 07:18:15.714768 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710705    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 07:18:15.714768 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710708    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 07:18:15.714768 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710711    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 07:18:15.714768 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710714    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 07:18:15.714768 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710716    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 07:18:15.714768 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710719    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 07:18:15.714768 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710722    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 07:18:15.714768 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710725    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 07:18:15.714768 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710728    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 07:18:15.714768 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710731    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 07:18:15.715187 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710733    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 07:18:15.715187 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710736    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 07:18:15.715187 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710738    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 07:18:15.715187 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710741    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 07:18:15.715187 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710744    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 07:18:15.715187 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710746    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 07:18:15.715187 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710749    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 07:18:15.715187 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710751    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 07:18:15.715187 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710754    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 07:18:15.715187 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710757    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 07:18:15.715187 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710759    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 07:18:15.715187 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710762    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 07:18:15.715187 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710765    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 07:18:15.715187 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710767    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 07:18:15.715187 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710770    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 07:18:15.715187 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710772    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 07:18:15.715187 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710775    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 07:18:15.715187 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710777    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 07:18:15.715187 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710780    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 07:18:15.715739 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710782    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 07:18:15.715739 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710785    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 07:18:15.715739 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710788    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 07:18:15.715739 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710790    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 07:18:15.715739 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710792    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 07:18:15.715739 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710795    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 07:18:15.715739 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710797    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 07:18:15.715739 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710800    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 07:18:15.715739 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710803    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 07:18:15.715739 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710805    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 07:18:15.715739 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710808    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 07:18:15.715739 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710810    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 07:18:15.715739 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710814    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 07:18:15.715739 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710816    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 07:18:15.715739 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710819    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 07:18:15.715739 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710836    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 07:18:15.715739 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710840    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 07:18:15.715739 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710843    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 07:18:15.715739 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710845    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 07:18:15.715739 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710848    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 07:18:15.716520 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710852    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 07:18:15.716520 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710855    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 07:18:15.716520 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710859    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 07:18:15.716520 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710862    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 07:18:15.716520 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710865    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 07:18:15.716520 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710868    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 07:18:15.716520 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710870    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 07:18:15.716520 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710873    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 07:18:15.716520 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710876    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 07:18:15.716520 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710878    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 07:18:15.716520 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710881    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 07:18:15.716520 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710884    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 07:18:15.716520 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710886    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 07:18:15.716520 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710889    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 07:18:15.716520 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710892    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 07:18:15.716520 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710895    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 07:18:15.716520 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710897    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 07:18:15.716520 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710900    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 07:18:15.716520 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710902    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 07:18:15.717189 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710905    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 07:18:15.717189 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710907    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 07:18:15.717189 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710910    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 07:18:15.717189 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710912    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 07:18:15.717189 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710915    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 07:18:15.717189 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710917    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 07:18:15.717189 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710920    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 07:18:15.717189 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710923    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 07:18:15.717189 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710926    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 07:18:15.717189 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710928    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 07:18:15.717189 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710931    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 07:18:15.717189 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710933    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 07:18:15.717189 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710936    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 07:18:15.717189 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710938    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 07:18:15.717189 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710942    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 07:18:15.717189 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710945    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 07:18:15.717189 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710948    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 07:18:15.717189 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710951    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 07:18:15.717189 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710955    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 07:18:15.717731 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710959    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 07:18:15.717731 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710962    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 07:18:15.717731 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710965    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 07:18:15.717731 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710967    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 07:18:15.717731 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710970    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 07:18:15.717731 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710972    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 07:18:15.717731 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710974    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 07:18:15.717731 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:15.710977    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 07:18:15.717731 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.710981    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 07:18:15.717731 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.711089    2571 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 07:18:15.717731 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.714181    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 07:18:15.717731 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.715020    2571 server.go:1019] "Starting client certificate rotation"
Sep 15 07:18:15.717731 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.715135    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 07:18:15.717731 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.715201    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 07:18:15.718139 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.718008    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 07:18:15.719131 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.719111    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 07:18:15.726164 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.726149    2571 log.go:25] "Validated CRI v1 runtime API"
Sep 15 07:18:15.729797 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.729778    2571 log.go:25] "Validated CRI v1 image API"
Sep 15 07:18:15.730755 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.730741    2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 07:18:15.733524 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.733504    2571 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 828b4280-41bb-4f65-9f2c-a22eb3c57511:/dev/nvme0n1p4 b7c75b6b-3437-45ad-bb08-d7e7b34029bb:/dev/nvme0n1p3]
Sep 15 07:18:15.733597 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.733522    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 07:18:15.738451 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.738337    2571 manager.go:216] Machine: {Timestamp:2026-09-15 07:18:15.73708934 +0000 UTC m=+0.283659450 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100011 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ee79847f8367fa337a9865c45120e SystemUUID:ec2ee798-47f8-367f-a337-a9865c45120e BootID:29773a2e-3009-49de-a2ea-8cfb9b625d2a Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ed:c6:99:4b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ed:c6:99:4b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:3b:7f:ec:7a:7e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 07:18:15.738451 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.738442    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 07:18:15.738604 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.738523    2571 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 07:18:15.741633 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.741604    2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 07:18:15.741764 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.741635    2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-76.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 07:18:15.742283 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.742273    2571 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 07:18:15.742321 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.742286    2571 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 07:18:15.742321 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.742294    2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 07:18:15.742562 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.742547    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 07:18:15.742851 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.742840    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 07:18:15.743569 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.743560    2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 07:18:15.746782 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.746770    2571 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 07:18:15.746898 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.746889    2571 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 07:18:15.746934 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.746920    2571 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 07:18:15.746934 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.746930    2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 07:18:15.746991 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.746945    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 07:18:15.746991 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.746954    2571 kubelet.go:398] "Adding apiserver pod source"
Sep 15 07:18:15.746991 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.746962    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 07:18:15.747304 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.747288    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 07:18:15.747361 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.747314    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 07:18:15.749342 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.749319    2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 07:18:15.751229 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.751213    2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 07:18:15.751300 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.751277    2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 07:18:15.752569 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.752558    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 07:18:15.752617 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.752576    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 07:18:15.752617 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.752582    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 07:18:15.752617 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.752589    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 07:18:15.752617 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.752601    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 07:18:15.752617 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.752610    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 07:18:15.752755 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.752623    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 07:18:15.752755 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.752629    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 07:18:15.752755 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.752636    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 07:18:15.752755 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.752642    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 07:18:15.752755 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.752650    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 07:18:15.752755 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.752655    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 07:18:15.753340 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.753330    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 07:18:15.753340 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.753339    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 07:18:15.756036 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.756016    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-76.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 07:18:15.756127 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.756091    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-76.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 07:18:15.756174 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.756128    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 07:18:15.757063 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.757048    2571 server.go:1268] "Started kubelet"
Sep 15 07:18:15.757143 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.757114    2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 07:18:15.757192 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.757141    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 07:18:15.757239 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.757205    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 07:18:15.758050 ip-10-0-129-76 systemd[1]: Started Kubernetes Kubelet.
Sep 15 07:18:15.758335 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.758272    2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 07:18:15.759137 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.759110    2571 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 07:18:15.761855 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.761810    2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 07:18:15.765247 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.764234    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-76.ec2.internal.18d56d62438aa09d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-76.ec2.internal,UID:ip-10-0-129-76.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-76.ec2.internal,},FirstTimestamp:2026-09-15 07:18:15.757021341 +0000 UTC m=+0.303591430,LastTimestamp:2026-09-15 07:18:15.757021341 +0000 UTC m=+0.303591430,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-76.ec2.internal,}"
Sep 15 07:18:15.765514 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.765496    2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 07:18:15.765623 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.765514    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 07:18:15.765623 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.765589    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-76.ec2.internal\" not found"
Sep 15 07:18:15.766228 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.766013    2571 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 07:18:15.766228 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.766030    2571 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 07:18:15.766228 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.766156    2571 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 07:18:15.766228 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.766165    2571 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 07:18:15.766459 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.766278    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 07:18:15.766459 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.766449    2571 factory.go:55] Registering systemd factory
Sep 15 07:18:15.766550 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.766463    2571 factory.go:223] Registration of the systemd container factory successfully
Sep 15 07:18:15.766687 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.766671    2571 factory.go:153] Registering CRI-O factory
Sep 15 07:18:15.766754 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.766692    2571 factory.go:223] Registration of the crio container factory successfully
Sep 15 07:18:15.766754 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.766751    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 07:18:15.766860 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.766772    2571 factory.go:103] Registering Raw factory
Sep 15 07:18:15.766860 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.766789    2571 manager.go:1196] Started watching for new ooms in manager
Sep 15 07:18:15.767232 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.767215    2571 manager.go:349] Starting recovery of all containers
Sep 15 07:18:15.769806 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.769779    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 15 07:18:15.769874 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.769851    2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-76.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 15 07:18:15.777373 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.777172    2571 manager.go:354] Recovery completed
Sep 15 07:18:15.781884 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.781863    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 07:18:15.783639 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.783613    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 07:18:15.783702 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.783653    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 07:18:15.783702 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.783667    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasSufficientPID"
Sep 15 07:18:15.784223 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.784210    2571 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 07:18:15.784223 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.784221    2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 07:18:15.784322 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.784236    2571 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 07:18:15.784322 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.784291    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 07:18:15.785033 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.785020    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 07:18:15.785033 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.785046    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 07:18:15.785153 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.785056    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasSufficientPID"
Sep 15 07:18:15.785302 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.785186    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-76.ec2.internal.18d56d624520c53f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-76.ec2.internal,UID:ip-10-0-129-76.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-76.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-76.ec2.internal,},FirstTimestamp:2026-09-15 07:18:15.783638335 +0000 UTC m=+0.330208424,LastTimestamp:2026-09-15 07:18:15.783638335 +0000 UTC m=+0.330208424,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-76.ec2.internal,}"
Sep 15 07:18:15.785432 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.785419    2571 policy_none.go:49] "None policy: Start"
Sep 15 07:18:15.785475 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.785438    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 07:18:15.785475 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.785452    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 07:18:15.786348 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.786336    2571 policy_none.go:47] "Start"
Sep 15 07:18:15.790118 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.790104    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ksbm6"
Sep 15 07:18:15.793859 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.793797    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-76.ec2.internal.18d56d62452116ca  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-76.ec2.internal,UID:ip-10-0-129-76.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-76.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-76.ec2.internal,},FirstTimestamp:2026-09-15 07:18:15.78365921 +0000 UTC m=+0.330229301,LastTimestamp:2026-09-15 07:18:15.78365921 +0000 UTC m=+0.330229301,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-76.ec2.internal,}"
Sep 15 07:18:15.796032 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.795968    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-76.ec2.internal.18d56d62452148b9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-76.ec2.internal,UID:ip-10-0-129-76.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-76.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-76.ec2.internal,},FirstTimestamp:2026-09-15 07:18:15.783671993 +0000 UTC m=+0.330242084,LastTimestamp:2026-09-15 07:18:15.783671993 +0000 UTC m=+0.330242084,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-76.ec2.internal,}"
Sep 15 07:18:15.798467 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.798331    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ksbm6"
Sep 15 07:18:15.803226 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.803150    2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-76.ec2.internal.18d56d624520c53f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-76.ec2.internal.18d56d624520c53f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-76.ec2.internal,UID:ip-10-0-129-76.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-76.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-76.ec2.internal,},FirstTimestamp:2026-09-15 07:18:15.783638335 +0000 UTC m=+0.330208424,LastTimestamp:2026-09-15 07:18:15.785032995 +0000 UTC m=+0.331603088,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-76.ec2.internal,}"
Sep 15 07:18:15.804993 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.804933    2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-76.ec2.internal.18d56d62452116ca\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-76.ec2.internal.18d56d62452116ca  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-76.ec2.internal,UID:ip-10-0-129-76.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-76.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-76.ec2.internal,},FirstTimestamp:2026-09-15 07:18:15.78365921 +0000 UTC m=+0.330229301,LastTimestamp:2026-09-15 07:18:15.785051192 +0000 UTC m=+0.331621282,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-76.ec2.internal,}"
Sep 15 07:18:15.811503 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.811448    2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-76.ec2.internal.18d56d62452148b9\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-76.ec2.internal.18d56d62452148b9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-76.ec2.internal,UID:ip-10-0-129-76.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-76.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-76.ec2.internal,},FirstTimestamp:2026-09-15 07:18:15.783671993 +0000 UTC m=+0.330242084,LastTimestamp:2026-09-15 07:18:15.785060368 +0000 UTC m=+0.331630461,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-76.ec2.internal,}"
Sep 15 07:18:15.814982 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.814961    2571 manager.go:337] "Starting Device Plugin manager"
Sep 15 07:18:15.815070 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.815002    2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 07:18:15.815070 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.815015    2571 server.go:87] "Starting device plugin registration server"
Sep 15 07:18:15.817568 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.815370    2571 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 07:18:15.817568 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.815382    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 07:18:15.817568 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.815522    2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 07:18:15.817568 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.815589    2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 07:18:15.817568 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.815597    2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 07:18:15.817568 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.816169    2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 07:18:15.817568 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.816222    2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-76.ec2.internal\" not found"
Sep 15 07:18:15.933422 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.819089    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-76.ec2.internal.18d56d624727415c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-76.ec2.internal,UID:ip-10-0-129-76.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-129-76.ec2.internal,},FirstTimestamp:2026-09-15 07:18:15.817617756 +0000 UTC m=+0.364187839,LastTimestamp:2026-09-15 07:18:15.817617756 +0000 UTC m=+0.364187839,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-76.ec2.internal,}"
Sep 15 07:18:15.933422 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.901021    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 07:18:15.933422 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.902201    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 07:18:15.933422 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.902276    2571 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 07:18:15.933422 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.902305    2571 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 07:18:15.933422 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.902401    2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 07:18:15.933422 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.911067    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 15 07:18:15.933422 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.916163    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 07:18:15.933422 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.917029    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 07:18:15.933422 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.917056    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 07:18:15.933422 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.917066    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasSufficientPID"
Sep 15 07:18:15.933422 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:15.917090    2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-76.ec2.internal"
Sep 15 07:18:15.933422 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.924950    2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-76.ec2.internal.18d56d624520c53f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-76.ec2.internal.18d56d624520c53f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-76.ec2.internal,UID:ip-10-0-129-76.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-76.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-76.ec2.internal,},FirstTimestamp:2026-09-15 07:18:15.783638335 +0000 UTC m=+0.330208424,LastTimestamp:2026-09-15 07:18:15.9170421 +0000 UTC m=+0.463612192,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-76.ec2.internal,}"
Sep 15 07:18:15.933422 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.928299    2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-129-76.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-129-76.ec2.internal"
Sep 15 07:18:15.933422 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.928331    2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-76.ec2.internal.18d56d62452116ca\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-76.ec2.internal.18d56d62452116ca  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-76.ec2.internal,UID:ip-10-0-129-76.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-76.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-76.ec2.internal,},FirstTimestamp:2026-09-15 07:18:15.78365921 +0000 UTC m=+0.330229301,LastTimestamp:2026-09-15 07:18:15.917060918 +0000 UTC m=+0.463631012,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-76.ec2.internal,}"
Sep 15 07:18:15.933422 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.932921    2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-76.ec2.internal.18d56d62452148b9\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-76.ec2.internal.18d56d62452148b9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-76.ec2.internal,UID:ip-10-0-129-76.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-76.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-76.ec2.internal,},FirstTimestamp:2026-09-15 07:18:15.783671993 +0000 UTC m=+0.330242084,LastTimestamp:2026-09-15 07:18:15.91707084 +0000 UTC m=+0.463640933,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-76.ec2.internal,}"
Sep 15 07:18:15.975818 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:15.975741    2571 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-76.ec2.internal\" not found" node="ip-10-0-129-76.ec2.internal"
Sep 15 07:18:16.002899 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.002849    2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-76.ec2.internal"]
Sep 15 07:18:16.002962 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.002947    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 07:18:16.003914 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.003898    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 07:18:16.004002 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.003926    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 07:18:16.004002 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.003936    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasSufficientPID"
Sep 15 07:18:16.005170 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.005158    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 07:18:16.005295 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.005270    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.ec2.internal"
Sep 15 07:18:16.005346 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.005314    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 07:18:16.005880 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.005865    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 07:18:16.005943 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.005894    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 07:18:16.005943 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.005903    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasSufficientPID"
Sep 15 07:18:16.006006 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.005866    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 07:18:16.006006 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.005968    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 07:18:16.006006 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.005978    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasSufficientPID"
Sep 15 07:18:16.007155 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.007138    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.ec2.internal"
Sep 15 07:18:16.007270 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.007168    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 07:18:16.007902 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.007884    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 07:18:16.007974 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.007912    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 07:18:16.007974 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.007927    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasSufficientPID"
Sep 15 07:18:16.036904 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:16.036882    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-76.ec2.internal\" not found" node="ip-10-0-129-76.ec2.internal"
Sep 15 07:18:16.041223 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:16.041207    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-76.ec2.internal\" not found" node="ip-10-0-129-76.ec2.internal"
Sep 15 07:18:16.068745 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.068723    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ed9aeb0260bd215aa885797c44981ced-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-76.ec2.internal\" (UID: \"ed9aeb0260bd215aa885797c44981ced\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.ec2.internal"
Sep 15 07:18:16.068745 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.068747    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed9aeb0260bd215aa885797c44981ced-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-76.ec2.internal\" (UID: \"ed9aeb0260bd215aa885797c44981ced\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.ec2.internal"
Sep 15 07:18:16.068873 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.068762    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/457fc68f7214b4c70c98b4a1d69ed2ba-config\") pod \"kube-apiserver-proxy-ip-10-0-129-76.ec2.internal\" (UID: \"457fc68f7214b4c70c98b4a1d69ed2ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.ec2.internal"
Sep 15 07:18:16.128886 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.128847    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 07:18:16.129866 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.129849    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 07:18:16.129954 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.129882    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 07:18:16.129954 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.129894    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasSufficientPID"
Sep 15 07:18:16.129954 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.129927    2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-76.ec2.internal"
Sep 15 07:18:16.139035 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.139021    2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-76.ec2.internal"
Sep 15 07:18:16.139102 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:16.139042    2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-76.ec2.internal\": node \"ip-10-0-129-76.ec2.internal\" not found"
Sep 15 07:18:16.169517 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.169494    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ed9aeb0260bd215aa885797c44981ced-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-76.ec2.internal\" (UID: \"ed9aeb0260bd215aa885797c44981ced\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.ec2.internal"
Sep 15 07:18:16.169517 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.169518    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed9aeb0260bd215aa885797c44981ced-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-76.ec2.internal\" (UID: \"ed9aeb0260bd215aa885797c44981ced\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.ec2.internal"
Sep 15 07:18:16.169635 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.169585    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/457fc68f7214b4c70c98b4a1d69ed2ba-config\") pod \"kube-apiserver-proxy-ip-10-0-129-76.ec2.internal\" (UID: \"457fc68f7214b4c70c98b4a1d69ed2ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.ec2.internal"
Sep 15 07:18:16.169635 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.169589    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ed9aeb0260bd215aa885797c44981ced-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-76.ec2.internal\" (UID: \"ed9aeb0260bd215aa885797c44981ced\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.ec2.internal"
Sep 15 07:18:16.169635 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.169589    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed9aeb0260bd215aa885797c44981ced-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-76.ec2.internal\" (UID: \"ed9aeb0260bd215aa885797c44981ced\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.ec2.internal"
Sep 15 07:18:16.169635 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.169612    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/457fc68f7214b4c70c98b4a1d69ed2ba-config\") pod \"kube-apiserver-proxy-ip-10-0-129-76.ec2.internal\" (UID: \"457fc68f7214b4c70c98b4a1d69ed2ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.ec2.internal"
Sep 15 07:18:16.194292 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:16.194247    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-76.ec2.internal\" not found"
Sep 15 07:18:16.295065 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:16.294995    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-76.ec2.internal\" not found"
Sep 15 07:18:16.339185 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.339154    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.ec2.internal"
Sep 15 07:18:16.344207 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.344182    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.ec2.internal"
Sep 15 07:18:16.395327 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:16.395297    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-76.ec2.internal\" not found"
Sep 15 07:18:16.495806 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:16.495765    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-76.ec2.internal\" not found"
Sep 15 07:18:16.597422 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:16.597346    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-76.ec2.internal\" not found"
Sep 15 07:18:16.698974 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:16.698939    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-76.ec2.internal\" not found"
Sep 15 07:18:16.715304 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.715283    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 07:18:16.800170 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:16.800132    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-76.ec2.internal\" not found"
Sep 15 07:18:16.894264 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:16.894222    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457fc68f7214b4c70c98b4a1d69ed2ba.slice/crio-a41446bc51ab5efe5288c8151908637bcf9bdb41154f4edc1d4e2353bfa2fa21 WatchSource:0}: Error finding container a41446bc51ab5efe5288c8151908637bcf9bdb41154f4edc1d4e2353bfa2fa21: Status 404 returned error can't find the container with id a41446bc51ab5efe5288c8151908637bcf9bdb41154f4edc1d4e2353bfa2fa21
Sep 15 07:18:16.894770 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:16.894739    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9aeb0260bd215aa885797c44981ced.slice/crio-77396814e0bff5521841a92628ccbfa74fbc20f800abdbdafbf458370bd46bc1 WatchSource:0}: Error finding container 77396814e0bff5521841a92628ccbfa74fbc20f800abdbdafbf458370bd46bc1: Status 404 returned error can't find the container with id 77396814e0bff5521841a92628ccbfa74fbc20f800abdbdafbf458370bd46bc1
Sep 15 07:18:16.899394 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.899372    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 07:18:16.901206 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:16.901186    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-76.ec2.internal\" not found"
Sep 15 07:18:16.905813 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.905770    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.ec2.internal" event={"ID":"ed9aeb0260bd215aa885797c44981ced","Type":"ContainerStarted","Data":"77396814e0bff5521841a92628ccbfa74fbc20f800abdbdafbf458370bd46bc1"}
Sep 15 07:18:16.906986 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.906966    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.ec2.internal" event={"ID":"457fc68f7214b4c70c98b4a1d69ed2ba","Type":"ContainerStarted","Data":"a41446bc51ab5efe5288c8151908637bcf9bdb41154f4edc1d4e2353bfa2fa21"}
Sep 15 07:18:16.928348 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.928326    2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 07:18:16.935451 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.935429    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 07:13:16 +0000 UTC" deadline="2028-04-13 02:37:13.134389917 +0000 UTC"
Sep 15 07:18:16.935451 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.935451    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13819h18m56.19894162s"
Sep 15 07:18:16.960512 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.960491    2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 07:18:16.987706 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.987678    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 07:18:16.988825 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.988808    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 07:18:16.988918 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.988838    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 07:18:16.988918 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:16.988850    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeHasSufficientPID"
Sep 15 07:18:17.002171 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:17.002144    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-76.ec2.internal\" not found"
Sep 15 07:18:17.104367 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:17.104334    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-76.ec2.internal\" not found"
Sep 15 07:18:17.204957 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:17.204870    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-76.ec2.internal\" not found"
Sep 15 07:18:17.307155 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:17.307120    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-76.ec2.internal\" not found"
Sep 15 07:18:17.358545 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.358520    2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 07:18:17.366831 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.366798    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.ec2.internal"
Sep 15 07:18:17.380194 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.380175    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 07:18:17.381082 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.381069    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.ec2.internal"
Sep 15 07:18:17.390089 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.390071    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 07:18:17.430149 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.430117    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 07:18:17.748698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.748662    2571 apiserver.go:52] "Watching apiserver"
Sep 15 07:18:17.756824 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.756797    2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 07:18:17.757312 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.757250    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-lkzcg","openshift-ovn-kubernetes/ovnkube-node-qtrll","kube-system/konnectivity-agent-dclwl","kube-system/kube-apiserver-proxy-ip-10-0-129-76.ec2.internal","openshift-cluster-node-tuning-operator/tuned-zg6jk","openshift-dns/node-resolver-zvc79","openshift-multus/multus-dbclm","openshift-multus/network-metrics-daemon-jhh6r","kube-system/global-pull-secret-syncer-kzskh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2","openshift-image-registry/node-ca-4scf4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.ec2.internal","openshift-multus/multus-additional-cni-plugins-r247m","openshift-network-diagnostics/network-check-target-rfcdm"]
Sep 15 07:18:17.759291 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.759249    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:17.760676 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.760653    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:17.761843 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.761778    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 07:18:17.762157 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.762127    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 07:18:17.762420 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.762402    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k42fj\""
Sep 15 07:18:17.762644 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.762628    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 07:18:17.766106 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.766082    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 07:18:17.766266 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.766244    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 07:18:17.766393 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.766372    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 07:18:17.766462 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.766393    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 07:18:17.766583 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.766566    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 07:18:17.766640 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.766626    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 07:18:17.766765 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.766749    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:17.766818 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.766767    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.766818 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.766767    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dclwl"
Sep 15 07:18:17.766818 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.766798    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-66pt5\""
Sep 15 07:18:17.766960 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:17.766839    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfcdm" podUID="7cc100f6-dbfe-412b-9ce2-845dea18988a"
Sep 15 07:18:17.767094 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.767075    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zvc79"
Sep 15 07:18:17.768571 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.768553    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:17.768665 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:17.768631    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzskh" podUID="08733b0b-6ce9-4bc5-87c5-4565bf0833b5"
Sep 15 07:18:17.769603 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.769582    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 07:18:17.769946 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.769929    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 07:18:17.770386 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.770366    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 07:18:17.770644 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.770625    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rrg7l\""
Sep 15 07:18:17.770713 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.770649    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 07:18:17.770897 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.770863    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.770897 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.770870    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vvrwr\""
Sep 15 07:18:17.771116 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.771064    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 07:18:17.771263 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.771248    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-llbfl\""
Sep 15 07:18:17.771378 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.771359    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 07:18:17.772065 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.772024    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.774562 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.774205    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lkzcg"
Sep 15 07:18:17.775243 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.775195    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 07:18:17.775460 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.775441    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4scf4"
Sep 15 07:18:17.775542 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.775491    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dp2bw\""
Sep 15 07:18:17.775869 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.775850    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2fvnh\""
Sep 15 07:18:17.775944 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.775910    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 07:18:17.776186 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.776169    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 07:18:17.776343 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.776314    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:17.776411 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:17.776391    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhh6r" podUID="bf0b2425-be3b-4095-b8ab-a44f426176ac"
Sep 15 07:18:17.776411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.776401    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 07:18:17.776516 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.776414    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 07:18:17.776516 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.776448    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 07:18:17.776807 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.776788    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 07:18:17.776878 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.776842    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 07:18:17.776955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.776938    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 07:18:17.777714 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.777697    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 07:18:17.777802 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.777790    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tk4jp\""
Sep 15 07:18:17.778462 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.778306    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 07:18:17.778698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.778680    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 07:18:17.778847 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.778689    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 07:18:17.778847 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.778843    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-g7dn9\""
Sep 15 07:18:17.779552 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.779530    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-tmp\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.783824 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.783800    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/061e1b77-3f2d-4f1a-b389-04ffdf5d0530-agent-certs\") pod \"konnectivity-agent-dclwl\" (UID: \"061e1b77-3f2d-4f1a-b389-04ffdf5d0530\") " pod="kube-system/konnectivity-agent-dclwl"
Sep 15 07:18:17.783934 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.783853    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nbrf\" (UniqueName: \"kubernetes.io/projected/a1932db5-d903-4b17-ac25-36613aa04ba4-kube-api-access-6nbrf\") pod \"node-resolver-zvc79\" (UID: \"a1932db5-d903-4b17-ac25-36613aa04ba4\") " pod="openshift-dns/node-resolver-zvc79"
Sep 15 07:18:17.783934 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.783888    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-kubelet-config\") pod \"global-pull-secret-syncer-kzskh\" (UID: \"08733b0b-6ce9-4bc5-87c5-4565bf0833b5\") " pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:17.784049 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.783930    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret\") pod \"global-pull-secret-syncer-kzskh\" (UID: \"08733b0b-6ce9-4bc5-87c5-4565bf0833b5\") " pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:17.784049 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.783958    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-etc-kubernetes\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.784049 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.783991    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwl8x\" (UniqueName: \"kubernetes.io/projected/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-kube-api-access-hwl8x\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.784049 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784025    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/542e179d-f430-4b8c-83f9-a7b45998ae91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r247m\" (UID: \"542e179d-f430-4b8c-83f9-a7b45998ae91\") " pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:17.784254 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784061    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-etc-modprobe-d\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.784254 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784092    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c5b21b3-5de6-4167-b785-fdf32d77e3af-device-dir\") pod \"aws-ebs-csi-driver-node-sdqf2\" (UID: \"8c5b21b3-5de6-4167-b785-fdf32d77e3af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:17.784254 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784125    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/061e1b77-3f2d-4f1a-b389-04ffdf5d0530-konnectivity-ca\") pod \"konnectivity-agent-dclwl\" (UID: \"061e1b77-3f2d-4f1a-b389-04ffdf5d0530\") " pod="kube-system/konnectivity-agent-dclwl"
Sep 15 07:18:17.784254 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784153    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-lib-modules\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.784254 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784185    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-var-lib-kubelet\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.784254 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784216    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f6ld\" (UniqueName: \"kubernetes.io/projected/8c5b21b3-5de6-4167-b785-fdf32d77e3af-kube-api-access-2f6ld\") pod \"aws-ebs-csi-driver-node-sdqf2\" (UID: \"8c5b21b3-5de6-4167-b785-fdf32d77e3af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:17.784254 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784249    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1932db5-d903-4b17-ac25-36613aa04ba4-tmp-dir\") pod \"node-resolver-zvc79\" (UID: \"a1932db5-d903-4b17-ac25-36613aa04ba4\") " pod="openshift-dns/node-resolver-zvc79"
Sep 15 07:18:17.784618 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784274    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-sys\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.784618 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784317    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-host\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.784618 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784357    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c5b21b3-5de6-4167-b785-fdf32d77e3af-socket-dir\") pod \"aws-ebs-csi-driver-node-sdqf2\" (UID: \"8c5b21b3-5de6-4167-b785-fdf32d77e3af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:17.784618 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784390    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c5b21b3-5de6-4167-b785-fdf32d77e3af-etc-selinux\") pod \"aws-ebs-csi-driver-node-sdqf2\" (UID: \"8c5b21b3-5de6-4167-b785-fdf32d77e3af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:17.784618 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784432    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/542e179d-f430-4b8c-83f9-a7b45998ae91-cni-binary-copy\") pod \"multus-additional-cni-plugins-r247m\" (UID: \"542e179d-f430-4b8c-83f9-a7b45998ae91\") " pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:17.784618 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784465    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/542e179d-f430-4b8c-83f9-a7b45998ae91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r247m\" (UID: \"542e179d-f430-4b8c-83f9-a7b45998ae91\") " pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:17.784618 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784494    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1932db5-d903-4b17-ac25-36613aa04ba4-hosts-file\") pod \"node-resolver-zvc79\" (UID: \"a1932db5-d903-4b17-ac25-36613aa04ba4\") " pod="openshift-dns/node-resolver-zvc79"
Sep 15 07:18:17.784618 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784525    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-etc-sysctl-d\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.784618 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784555    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-etc-systemd\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.784618 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784586    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-etc-tuned\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.784618 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784616    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c5b21b3-5de6-4167-b785-fdf32d77e3af-sys-fs\") pod \"aws-ebs-csi-driver-node-sdqf2\" (UID: \"8c5b21b3-5de6-4167-b785-fdf32d77e3af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:17.785193 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784653    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/542e179d-f430-4b8c-83f9-a7b45998ae91-system-cni-dir\") pod \"multus-additional-cni-plugins-r247m\" (UID: \"542e179d-f430-4b8c-83f9-a7b45998ae91\") " pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:17.785193 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784693    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/542e179d-f430-4b8c-83f9-a7b45998ae91-cnibin\") pod \"multus-additional-cni-plugins-r247m\" (UID: \"542e179d-f430-4b8c-83f9-a7b45998ae91\") " pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:17.785193 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784725    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-dbus\") pod \"global-pull-secret-syncer-kzskh\" (UID: \"08733b0b-6ce9-4bc5-87c5-4565bf0833b5\") " pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:17.785193 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784759    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c5b21b3-5de6-4167-b785-fdf32d77e3af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sdqf2\" (UID: \"8c5b21b3-5de6-4167-b785-fdf32d77e3af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:17.785193 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784789    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c5b21b3-5de6-4167-b785-fdf32d77e3af-registration-dir\") pod \"aws-ebs-csi-driver-node-sdqf2\" (UID: \"8c5b21b3-5de6-4167-b785-fdf32d77e3af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:17.785193 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784824    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/542e179d-f430-4b8c-83f9-a7b45998ae91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r247m\" (UID: \"542e179d-f430-4b8c-83f9-a7b45998ae91\") " pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:17.785193 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784859    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhk4j\" (UniqueName: \"kubernetes.io/projected/542e179d-f430-4b8c-83f9-a7b45998ae91-kube-api-access-jhk4j\") pod \"multus-additional-cni-plugins-r247m\" (UID: \"542e179d-f430-4b8c-83f9-a7b45998ae91\") " pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:17.785193 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784893    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5kzw\" (UniqueName: \"kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw\") pod \"network-check-target-rfcdm\" (UID: \"7cc100f6-dbfe-412b-9ce2-845dea18988a\") " pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:17.785193 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784935    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-etc-sysconfig\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.785193 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784967    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-etc-sysctl-conf\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.785193 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.784995    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-run\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.787141 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.787118    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 07:18:17.801869 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.801844    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nqf5k"
Sep 15 07:18:17.810883 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.810844    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nqf5k"
Sep 15 07:18:17.867971 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.867932    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 07:18:17.885577 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.885541    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-multus-socket-dir-parent\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.885772 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.885591    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-etc-sysctl-d\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.885772 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.885621    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-etc-systemd\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.885772 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.885646    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/542e179d-f430-4b8c-83f9-a7b45998ae91-cnibin\") pod \"multus-additional-cni-plugins-r247m\" (UID: \"542e179d-f430-4b8c-83f9-a7b45998ae91\") " pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:17.885772 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.885674    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-run-openvswitch\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.885772 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.885700    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.885772 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.885726    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-host-cni-netd\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.885772 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.885747    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-dbus\") pod \"global-pull-secret-syncer-kzskh\" (UID: \"08733b0b-6ce9-4bc5-87c5-4565bf0833b5\") " pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:17.886144 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.885781    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c5b21b3-5de6-4167-b785-fdf32d77e3af-registration-dir\") pod \"aws-ebs-csi-driver-node-sdqf2\" (UID: \"8c5b21b3-5de6-4167-b785-fdf32d77e3af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:17.886144 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.885799    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/542e179d-f430-4b8c-83f9-a7b45998ae91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r247m\" (UID: \"542e179d-f430-4b8c-83f9-a7b45998ae91\") " pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:17.886144 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.885819    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/850e8885-aca8-4688-8f07-087daf128b3d-multus-daemon-config\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.886144 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.885841    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-var-lib-openvswitch\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.886144 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.885858    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5a87606-6562-4269-b2be-0618e3d86409-ovnkube-config\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.886144 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.885891    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-cnibin\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.886144 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.885910    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-run-systemd\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.886144 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.885927    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njr82\" (UniqueName: \"kubernetes.io/projected/7ecf978c-c2fb-4db3-a60c-d6d4eb3d7f50-kube-api-access-njr82\") pod \"iptables-alerter-lkzcg\" (UID: \"7ecf978c-c2fb-4db3-a60c-d6d4eb3d7f50\") " pod="openshift-network-operator/iptables-alerter-lkzcg"
Sep 15 07:18:17.886144 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.885944    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-kubelet-config\") pod \"global-pull-secret-syncer-kzskh\" (UID: \"08733b0b-6ce9-4bc5-87c5-4565bf0833b5\") " pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:17.886144 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.885959    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret\") pod \"global-pull-secret-syncer-kzskh\" (UID: \"08733b0b-6ce9-4bc5-87c5-4565bf0833b5\") " pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:17.886144 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.885975    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwl8x\" (UniqueName: \"kubernetes.io/projected/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-kube-api-access-hwl8x\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.886144 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.885990    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-system-cni-dir\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.886144 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886007    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9j8w\" (UniqueName: \"kubernetes.io/projected/850e8885-aca8-4688-8f07-087daf128b3d-kube-api-access-l9j8w\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.886144 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886025    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-systemd-units\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.886144 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886039    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-host-run-netns\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.886144 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886054    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5a87606-6562-4269-b2be-0618e3d86409-ovn-node-metrics-cert\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.886889 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886068    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7ecf978c-c2fb-4db3-a60c-d6d4eb3d7f50-iptables-alerter-script\") pod \"iptables-alerter-lkzcg\" (UID: \"7ecf978c-c2fb-4db3-a60c-d6d4eb3d7f50\") " pod="openshift-network-operator/iptables-alerter-lkzcg"
Sep 15 07:18:17.886889 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886083    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c38076ad-5b5b-4fff-8ad9-149153f9d9d2-host\") pod \"node-ca-4scf4\" (UID: \"c38076ad-5b5b-4fff-8ad9-149153f9d9d2\") " pod="openshift-image-registry/node-ca-4scf4"
Sep 15 07:18:17.886889 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886105    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-var-lib-kubelet\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.886889 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886121    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f6ld\" (UniqueName: \"kubernetes.io/projected/8c5b21b3-5de6-4167-b785-fdf32d77e3af-kube-api-access-2f6ld\") pod \"aws-ebs-csi-driver-node-sdqf2\" (UID: \"8c5b21b3-5de6-4167-b785-fdf32d77e3af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:17.886889 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886137    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs\") pod \"network-metrics-daemon-jhh6r\" (UID: \"bf0b2425-be3b-4095-b8ab-a44f426176ac\") " pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:17.886889 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886153    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1932db5-d903-4b17-ac25-36613aa04ba4-tmp-dir\") pod \"node-resolver-zvc79\" (UID: \"a1932db5-d903-4b17-ac25-36613aa04ba4\") " pod="openshift-dns/node-resolver-zvc79"
Sep 15 07:18:17.886889 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886174    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-host-var-lib-cni-multus\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.886889 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886190    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5a87606-6562-4269-b2be-0618e3d86409-env-overrides\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.886889 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886206    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfcws\" (UniqueName: \"kubernetes.io/projected/b5a87606-6562-4269-b2be-0618e3d86409-kube-api-access-vfcws\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.886889 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886228    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-host-var-lib-kubelet\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.886889 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886285    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-hostroot\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.886889 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886317    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-multus-conf-dir\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.886889 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886349    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-kubelet-config\") pod \"global-pull-secret-syncer-kzskh\" (UID: \"08733b0b-6ce9-4bc5-87c5-4565bf0833b5\") " pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:17.886889 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886360    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-dbus\") pod \"global-pull-secret-syncer-kzskh\" (UID: \"08733b0b-6ce9-4bc5-87c5-4565bf0833b5\") " pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:17.886889 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886377    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c5b21b3-5de6-4167-b785-fdf32d77e3af-registration-dir\") pod \"aws-ebs-csi-driver-node-sdqf2\" (UID: \"8c5b21b3-5de6-4167-b785-fdf32d77e3af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:17.886889 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886450    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/542e179d-f430-4b8c-83f9-a7b45998ae91-cni-binary-copy\") pod \"multus-additional-cni-plugins-r247m\" (UID: \"542e179d-f430-4b8c-83f9-a7b45998ae91\") " pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:17.886889 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886499    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/542e179d-f430-4b8c-83f9-a7b45998ae91-cnibin\") pod \"multus-additional-cni-plugins-r247m\" (UID: \"542e179d-f430-4b8c-83f9-a7b45998ae91\") " pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:17.887770 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886513    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-var-lib-kubelet\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.887770 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886553    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-etc-systemd\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.887770 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886662    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-etc-sysctl-d\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.887770 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886754    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/542e179d-f430-4b8c-83f9-a7b45998ae91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r247m\" (UID: \"542e179d-f430-4b8c-83f9-a7b45998ae91\") " pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:17.887770 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.886760    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1932db5-d903-4b17-ac25-36613aa04ba4-tmp-dir\") pod \"node-resolver-zvc79\" (UID: \"a1932db5-d903-4b17-ac25-36613aa04ba4\") " pod="openshift-dns/node-resolver-zvc79"
Sep 15 07:18:17.887770 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:17.886987    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:17.887770 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887007    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-multus-cni-dir\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.887770 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:17.887073    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret podName:08733b0b-6ce9-4bc5-87c5-4565bf0833b5 nodeName:}" failed. No retries permitted until 2026-09-15 07:18:18.387051349 +0000 UTC m=+2.933621444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret") pod "global-pull-secret-syncer-kzskh" (UID: "08733b0b-6ce9-4bc5-87c5-4565bf0833b5") : object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:17.887770 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887090    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/542e179d-f430-4b8c-83f9-a7b45998ae91-cni-binary-copy\") pod \"multus-additional-cni-plugins-r247m\" (UID: \"542e179d-f430-4b8c-83f9-a7b45998ae91\") " pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:17.887770 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887096    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-host-var-lib-cni-bin\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.887770 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887157    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-host-cni-bin\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.887770 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887187    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-etc-kubernetes\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.887770 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887217    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1932db5-d903-4b17-ac25-36613aa04ba4-hosts-file\") pod \"node-resolver-zvc79\" (UID: \"a1932db5-d903-4b17-ac25-36613aa04ba4\") " pod="openshift-dns/node-resolver-zvc79"
Sep 15 07:18:17.887770 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887244    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-etc-tuned\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.887770 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887271    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c5b21b3-5de6-4167-b785-fdf32d77e3af-sys-fs\") pod \"aws-ebs-csi-driver-node-sdqf2\" (UID: \"8c5b21b3-5de6-4167-b785-fdf32d77e3af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:17.887770 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887297    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/542e179d-f430-4b8c-83f9-a7b45998ae91-system-cni-dir\") pod \"multus-additional-cni-plugins-r247m\" (UID: \"542e179d-f430-4b8c-83f9-a7b45998ae91\") " pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:17.887770 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887310    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1932db5-d903-4b17-ac25-36613aa04ba4-hosts-file\") pod \"node-resolver-zvc79\" (UID: \"a1932db5-d903-4b17-ac25-36613aa04ba4\") " pod="openshift-dns/node-resolver-zvc79"
Sep 15 07:18:17.888656 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887328    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ecf978c-c2fb-4db3-a60c-d6d4eb3d7f50-host-slash\") pod \"iptables-alerter-lkzcg\" (UID: \"7ecf978c-c2fb-4db3-a60c-d6d4eb3d7f50\") " pod="openshift-network-operator/iptables-alerter-lkzcg"
Sep 15 07:18:17.888656 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887376    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c5b21b3-5de6-4167-b785-fdf32d77e3af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sdqf2\" (UID: \"8c5b21b3-5de6-4167-b785-fdf32d77e3af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:17.888656 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887401    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhk4j\" (UniqueName: \"kubernetes.io/projected/542e179d-f430-4b8c-83f9-a7b45998ae91-kube-api-access-jhk4j\") pod \"multus-additional-cni-plugins-r247m\" (UID: \"542e179d-f430-4b8c-83f9-a7b45998ae91\") " pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:17.888656 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887416    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c5b21b3-5de6-4167-b785-fdf32d77e3af-sys-fs\") pod \"aws-ebs-csi-driver-node-sdqf2\" (UID: \"8c5b21b3-5de6-4167-b785-fdf32d77e3af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:17.888656 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887430    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-host-run-netns\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.888656 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887460    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5kzw\" (UniqueName: \"kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw\") pod \"network-check-target-rfcdm\" (UID: \"7cc100f6-dbfe-412b-9ce2-845dea18988a\") " pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:17.888656 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887480    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/542e179d-f430-4b8c-83f9-a7b45998ae91-system-cni-dir\") pod \"multus-additional-cni-plugins-r247m\" (UID: \"542e179d-f430-4b8c-83f9-a7b45998ae91\") " pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:17.888656 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887485    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-etc-sysconfig\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.888656 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887518    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-etc-sysctl-conf\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.888656 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887534    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-run\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.888656 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887540    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-etc-sysconfig\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.888656 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887548    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-tmp\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.888656 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887578    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-host-kubelet\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.888656 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887607    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.888656 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887637    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/061e1b77-3f2d-4f1a-b389-04ffdf5d0530-agent-certs\") pod \"konnectivity-agent-dclwl\" (UID: \"061e1b77-3f2d-4f1a-b389-04ffdf5d0530\") " pod="kube-system/konnectivity-agent-dclwl"
Sep 15 07:18:17.888656 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887665    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nbrf\" (UniqueName: \"kubernetes.io/projected/a1932db5-d903-4b17-ac25-36613aa04ba4-kube-api-access-6nbrf\") pod \"node-resolver-zvc79\" (UID: \"a1932db5-d903-4b17-ac25-36613aa04ba4\") " pod="openshift-dns/node-resolver-zvc79"
Sep 15 07:18:17.888656 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887681    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-run\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.889404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887692    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-etc-kubernetes\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.889404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887719    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/542e179d-f430-4b8c-83f9-a7b45998ae91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r247m\" (UID: \"542e179d-f430-4b8c-83f9-a7b45998ae91\") " pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:17.889404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887750    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq72j\" (UniqueName: \"kubernetes.io/projected/bf0b2425-be3b-4095-b8ab-a44f426176ac-kube-api-access-dq72j\") pod \"network-metrics-daemon-jhh6r\" (UID: \"bf0b2425-be3b-4095-b8ab-a44f426176ac\") " pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:17.889404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887776    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-etc-modprobe-d\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.889404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887801    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c5b21b3-5de6-4167-b785-fdf32d77e3af-device-dir\") pod \"aws-ebs-csi-driver-node-sdqf2\" (UID: \"8c5b21b3-5de6-4167-b785-fdf32d77e3af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:17.889404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887829    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/850e8885-aca8-4688-8f07-087daf128b3d-cni-binary-copy\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.889404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887723    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c5b21b3-5de6-4167-b785-fdf32d77e3af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sdqf2\" (UID: \"8c5b21b3-5de6-4167-b785-fdf32d77e3af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:17.889404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887951    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-etc-kubernetes\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.889404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887984    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-host-slash\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.889404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.887970    2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 07:18:17.889404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888014    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-log-socket\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.889404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888026    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c5b21b3-5de6-4167-b785-fdf32d77e3af-device-dir\") pod \"aws-ebs-csi-driver-node-sdqf2\" (UID: \"8c5b21b3-5de6-4167-b785-fdf32d77e3af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:17.889404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888044    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5a87606-6562-4269-b2be-0618e3d86409-ovnkube-script-lib\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.889404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888057    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-etc-sysctl-conf\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.889404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888082    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c38076ad-5b5b-4fff-8ad9-149153f9d9d2-serviceca\") pod \"node-ca-4scf4\" (UID: \"c38076ad-5b5b-4fff-8ad9-149153f9d9d2\") " pod="openshift-image-registry/node-ca-4scf4"
Sep 15 07:18:17.889404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888134    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/061e1b77-3f2d-4f1a-b389-04ffdf5d0530-konnectivity-ca\") pod \"konnectivity-agent-dclwl\" (UID: \"061e1b77-3f2d-4f1a-b389-04ffdf5d0530\") " pod="kube-system/konnectivity-agent-dclwl"
Sep 15 07:18:17.889404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888160    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-lib-modules\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.890188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888185    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-etc-openvswitch\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.890188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888211    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-run-ovn\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.890188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888236    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-node-log\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.890188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888299    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-sys\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.890188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888330    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-host\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.890188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888357    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/542e179d-f430-4b8c-83f9-a7b45998ae91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r247m\" (UID: \"542e179d-f430-4b8c-83f9-a7b45998ae91\") " pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:17.890188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888375    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-host-run-k8s-cni-cncf-io\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.890188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888408    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpcfh\" (UniqueName: \"kubernetes.io/projected/c38076ad-5b5b-4fff-8ad9-149153f9d9d2-kube-api-access-xpcfh\") pod \"node-ca-4scf4\" (UID: \"c38076ad-5b5b-4fff-8ad9-149153f9d9d2\") " pod="openshift-image-registry/node-ca-4scf4"
Sep 15 07:18:17.890188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888443    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-host-run-multus-certs\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.890188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888486    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-etc-modprobe-d\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.890188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888583    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-lib-modules\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.890188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888631    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c5b21b3-5de6-4167-b785-fdf32d77e3af-socket-dir\") pod \"aws-ebs-csi-driver-node-sdqf2\" (UID: \"8c5b21b3-5de6-4167-b785-fdf32d77e3af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:17.890188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888682    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-host\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.890188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888707    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c5b21b3-5de6-4167-b785-fdf32d77e3af-etc-selinux\") pod \"aws-ebs-csi-driver-node-sdqf2\" (UID: \"8c5b21b3-5de6-4167-b785-fdf32d77e3af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:17.890188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888710    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-sys\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.890188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888736    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/542e179d-f430-4b8c-83f9-a7b45998ae91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r247m\" (UID: \"542e179d-f430-4b8c-83f9-a7b45998ae91\") " pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:17.890188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888779    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/061e1b77-3f2d-4f1a-b389-04ffdf5d0530-konnectivity-ca\") pod \"konnectivity-agent-dclwl\" (UID: \"061e1b77-3f2d-4f1a-b389-04ffdf5d0530\") " pod="kube-system/konnectivity-agent-dclwl"
Sep 15 07:18:17.891103 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888784    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c5b21b3-5de6-4167-b785-fdf32d77e3af-etc-selinux\") pod \"aws-ebs-csi-driver-node-sdqf2\" (UID: \"8c5b21b3-5de6-4167-b785-fdf32d77e3af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:17.891103 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.888863    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c5b21b3-5de6-4167-b785-fdf32d77e3af-socket-dir\") pod \"aws-ebs-csi-driver-node-sdqf2\" (UID: \"8c5b21b3-5de6-4167-b785-fdf32d77e3af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:17.891103 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.889232    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/542e179d-f430-4b8c-83f9-a7b45998ae91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r247m\" (UID: \"542e179d-f430-4b8c-83f9-a7b45998ae91\") " pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:17.892527 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.892502    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-tmp\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.892650 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.892553    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-etc-tuned\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.892848 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.892819    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/061e1b77-3f2d-4f1a-b389-04ffdf5d0530-agent-certs\") pod \"konnectivity-agent-dclwl\" (UID: \"061e1b77-3f2d-4f1a-b389-04ffdf5d0530\") " pod="kube-system/konnectivity-agent-dclwl"
Sep 15 07:18:17.899135 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.899038    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f6ld\" (UniqueName: \"kubernetes.io/projected/8c5b21b3-5de6-4167-b785-fdf32d77e3af-kube-api-access-2f6ld\") pod \"aws-ebs-csi-driver-node-sdqf2\" (UID: \"8c5b21b3-5de6-4167-b785-fdf32d77e3af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:17.899801 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.899736    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhk4j\" (UniqueName: \"kubernetes.io/projected/542e179d-f430-4b8c-83f9-a7b45998ae91-kube-api-access-jhk4j\") pod \"multus-additional-cni-plugins-r247m\" (UID: \"542e179d-f430-4b8c-83f9-a7b45998ae91\") " pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:17.899929 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:17.899872    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 07:18:17.899929 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:17.899893    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 07:18:17.899929 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:17.899907    2571 projected.go:196] Error preparing data for projected volume kube-api-access-s5kzw for pod openshift-network-diagnostics/network-check-target-rfcdm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:17.900091 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:17.899973    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw podName:7cc100f6-dbfe-412b-9ce2-845dea18988a nodeName:}" failed. No retries permitted until 2026-09-15 07:18:18.39995484 +0000 UTC m=+2.946524925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s5kzw" (UniqueName: "kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw") pod "network-check-target-rfcdm" (UID: "7cc100f6-dbfe-412b-9ce2-845dea18988a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:17.902119 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.902092    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nbrf\" (UniqueName: \"kubernetes.io/projected/a1932db5-d903-4b17-ac25-36613aa04ba4-kube-api-access-6nbrf\") pod \"node-resolver-zvc79\" (UID: \"a1932db5-d903-4b17-ac25-36613aa04ba4\") " pod="openshift-dns/node-resolver-zvc79"
Sep 15 07:18:17.905199 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.904975    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwl8x\" (UniqueName: \"kubernetes.io/projected/fc7833b4-e17c-4bc7-bd90-f293e9d38e15-kube-api-access-hwl8x\") pod \"tuned-zg6jk\" (UID: \"fc7833b4-e17c-4bc7-bd90-f293e9d38e15\") " pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:17.989468 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989433    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-host-kubelet\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.989468 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989475    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.989705 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989504    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq72j\" (UniqueName: \"kubernetes.io/projected/bf0b2425-be3b-4095-b8ab-a44f426176ac-kube-api-access-dq72j\") pod \"network-metrics-daemon-jhh6r\" (UID: \"bf0b2425-be3b-4095-b8ab-a44f426176ac\") " pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:17.989705 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989532    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/850e8885-aca8-4688-8f07-087daf128b3d-cni-binary-copy\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.989705 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989545    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-host-kubelet\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.989705 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989557    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-host-slash\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.989705 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989581    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.989705 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989593    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-log-socket\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.989705 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989621    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-log-socket\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.989705 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989654    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5a87606-6562-4269-b2be-0618e3d86409-ovnkube-script-lib\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.989705 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989686    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c38076ad-5b5b-4fff-8ad9-149153f9d9d2-serviceca\") pod \"node-ca-4scf4\" (UID: \"c38076ad-5b5b-4fff-8ad9-149153f9d9d2\") " pod="openshift-image-registry/node-ca-4scf4"
Sep 15 07:18:17.990188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989714    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-etc-openvswitch\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.990188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989735    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-run-ovn\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.990188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989738    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-host-slash\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.990188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989750    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-node-log\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.990188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989782    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-node-log\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.990188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989811    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-etc-openvswitch\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.990188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989848    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-host-run-k8s-cni-cncf-io\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.990188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989868    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpcfh\" (UniqueName: \"kubernetes.io/projected/c38076ad-5b5b-4fff-8ad9-149153f9d9d2-kube-api-access-xpcfh\") pod \"node-ca-4scf4\" (UID: \"c38076ad-5b5b-4fff-8ad9-149153f9d9d2\") " pod="openshift-image-registry/node-ca-4scf4"
Sep 15 07:18:17.990188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989892    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-host-run-multus-certs\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.990188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989916    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-multus-socket-dir-parent\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.990188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989940    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-run-openvswitch\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.990188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989956    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.990188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989980    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-host-cni-netd\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.990188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989998    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/850e8885-aca8-4688-8f07-087daf128b3d-multus-daemon-config\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.990188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990014    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-var-lib-openvswitch\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.990188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990037    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5a87606-6562-4269-b2be-0618e3d86409-ovnkube-config\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.990188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990060    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-cnibin\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.990188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990084    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-run-systemd\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.991252 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990100    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njr82\" (UniqueName: \"kubernetes.io/projected/7ecf978c-c2fb-4db3-a60c-d6d4eb3d7f50-kube-api-access-njr82\") pod \"iptables-alerter-lkzcg\" (UID: \"7ecf978c-c2fb-4db3-a60c-d6d4eb3d7f50\") " pod="openshift-network-operator/iptables-alerter-lkzcg"
Sep 15 07:18:17.991252 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990126    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-system-cni-dir\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.991252 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990141    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9j8w\" (UniqueName: \"kubernetes.io/projected/850e8885-aca8-4688-8f07-087daf128b3d-kube-api-access-l9j8w\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.991252 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990159    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/850e8885-aca8-4688-8f07-087daf128b3d-cni-binary-copy\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.991252 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990161    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c38076ad-5b5b-4fff-8ad9-149153f9d9d2-serviceca\") pod \"node-ca-4scf4\" (UID: \"c38076ad-5b5b-4fff-8ad9-149153f9d9d2\") " pod="openshift-image-registry/node-ca-4scf4"
Sep 15 07:18:17.991252 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990206    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-host-cni-netd\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.991252 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990163    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-systemd-units\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.991252 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990226    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-systemd-units\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.991252 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.989818    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-run-ovn\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.991252 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990260    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-host-run-k8s-cni-cncf-io\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.991252 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990290    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-host-run-netns\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.991252 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990349    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5a87606-6562-4269-b2be-0618e3d86409-ovn-node-metrics-cert\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.991252 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990377    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7ecf978c-c2fb-4db3-a60c-d6d4eb3d7f50-iptables-alerter-script\") pod \"iptables-alerter-lkzcg\" (UID: \"7ecf978c-c2fb-4db3-a60c-d6d4eb3d7f50\") " pod="openshift-network-operator/iptables-alerter-lkzcg"
Sep 15 07:18:17.991252 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990403    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c38076ad-5b5b-4fff-8ad9-149153f9d9d2-host\") pod \"node-ca-4scf4\" (UID: \"c38076ad-5b5b-4fff-8ad9-149153f9d9d2\") " pod="openshift-image-registry/node-ca-4scf4"
Sep 15 07:18:17.991252 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990447    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs\") pod \"network-metrics-daemon-jhh6r\" (UID: \"bf0b2425-be3b-4095-b8ab-a44f426176ac\") " pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:17.991252 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990467    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5a87606-6562-4269-b2be-0618e3d86409-ovnkube-script-lib\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.991252 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990523    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-cnibin\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.991252 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990527    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-multus-socket-dir-parent\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.991999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990567    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.991999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990609    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-run-systemd\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.991999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990602    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-run-openvswitch\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.991999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990613    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5a87606-6562-4269-b2be-0618e3d86409-ovnkube-config\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.991999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990641    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-var-lib-openvswitch\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.991999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990664    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-system-cni-dir\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.991999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990473    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-host-var-lib-cni-multus\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.991999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990729    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5a87606-6562-4269-b2be-0618e3d86409-env-overrides\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.991999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990760    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfcws\" (UniqueName: \"kubernetes.io/projected/b5a87606-6562-4269-b2be-0618e3d86409-kube-api-access-vfcws\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.991999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990789    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-host-var-lib-kubelet\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.991999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990815    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-hostroot\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.991999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990840    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-multus-conf-dir\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.991999 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:17.990868    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:17.991999 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:17.990916    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs podName:bf0b2425-be3b-4095-b8ab-a44f426176ac nodeName:}" failed. No retries permitted until 2026-09-15 07:18:18.490902653 +0000 UTC m=+3.037472732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs") pod "network-metrics-daemon-jhh6r" (UID: "bf0b2425-be3b-4095-b8ab-a44f426176ac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:17.991999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990933    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-multus-cni-dir\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.991999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990983    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-host-run-multus-certs\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.991999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.991059    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c38076ad-5b5b-4fff-8ad9-149153f9d9d2-host\") pod \"node-ca-4scf4\" (UID: \"c38076ad-5b5b-4fff-8ad9-149153f9d9d2\") " pod="openshift-image-registry/node-ca-4scf4"
Sep 15 07:18:17.991999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990869    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-multus-cni-dir\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.992647 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.991098    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-host-var-lib-cni-bin\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.992647 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.991121    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-host-cni-bin\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.992647 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.991126    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/850e8885-aca8-4688-8f07-087daf128b3d-multus-daemon-config\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.992647 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.991141    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-etc-kubernetes\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.992647 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.990567    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-host-var-lib-cni-multus\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.992647 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.991169    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7ecf978c-c2fb-4db3-a60c-d6d4eb3d7f50-iptables-alerter-script\") pod \"iptables-alerter-lkzcg\" (UID: \"7ecf978c-c2fb-4db3-a60c-d6d4eb3d7f50\") " pod="openshift-network-operator/iptables-alerter-lkzcg"
Sep 15 07:18:17.992647 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.991178    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ecf978c-c2fb-4db3-a60c-d6d4eb3d7f50-host-slash\") pod \"iptables-alerter-lkzcg\" (UID: \"7ecf978c-c2fb-4db3-a60c-d6d4eb3d7f50\") " pod="openshift-network-operator/iptables-alerter-lkzcg"
Sep 15 07:18:17.992647 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.991183    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-host-var-lib-cni-bin\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.992647 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.991212    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-host-cni-bin\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.992647 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.991220    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-host-run-netns\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.992647 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.991225    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5a87606-6562-4269-b2be-0618e3d86409-host-run-netns\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.992647 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.991242    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-etc-kubernetes\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.992647 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.991304    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-multus-conf-dir\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.992647 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.991309    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ecf978c-c2fb-4db3-a60c-d6d4eb3d7f50-host-slash\") pod \"iptables-alerter-lkzcg\" (UID: \"7ecf978c-c2fb-4db3-a60c-d6d4eb3d7f50\") " pod="openshift-network-operator/iptables-alerter-lkzcg"
Sep 15 07:18:17.992647 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.991359    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-host-run-netns\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.992647 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.991402    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-hostroot\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.992647 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.991416    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5a87606-6562-4269-b2be-0618e3d86409-env-overrides\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:17.992647 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.991483    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/850e8885-aca8-4688-8f07-087daf128b3d-host-var-lib-kubelet\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:17.993630 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:17.993605    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5a87606-6562-4269-b2be-0618e3d86409-ovn-node-metrics-cert\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:18.006079 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:18.006016    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpcfh\" (UniqueName: \"kubernetes.io/projected/c38076ad-5b5b-4fff-8ad9-149153f9d9d2-kube-api-access-xpcfh\") pod \"node-ca-4scf4\" (UID: \"c38076ad-5b5b-4fff-8ad9-149153f9d9d2\") " pod="openshift-image-registry/node-ca-4scf4"
Sep 15 07:18:18.006200 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:18.006181    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfcws\" (UniqueName: \"kubernetes.io/projected/b5a87606-6562-4269-b2be-0618e3d86409-kube-api-access-vfcws\") pod \"ovnkube-node-qtrll\" (UID: \"b5a87606-6562-4269-b2be-0618e3d86409\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:18.006906 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:18.006879    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq72j\" (UniqueName: \"kubernetes.io/projected/bf0b2425-be3b-4095-b8ab-a44f426176ac-kube-api-access-dq72j\") pod \"network-metrics-daemon-jhh6r\" (UID: \"bf0b2425-be3b-4095-b8ab-a44f426176ac\") " pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:18.007771 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:18.007751    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njr82\" (UniqueName: \"kubernetes.io/projected/7ecf978c-c2fb-4db3-a60c-d6d4eb3d7f50-kube-api-access-njr82\") pod \"iptables-alerter-lkzcg\" (UID: \"7ecf978c-c2fb-4db3-a60c-d6d4eb3d7f50\") " pod="openshift-network-operator/iptables-alerter-lkzcg"
Sep 15 07:18:18.008156 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:18.008138    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9j8w\" (UniqueName: \"kubernetes.io/projected/850e8885-aca8-4688-8f07-087daf128b3d-kube-api-access-l9j8w\") pod \"multus-dbclm\" (UID: \"850e8885-aca8-4688-8f07-087daf128b3d\") " pod="openshift-multus/multus-dbclm"
Sep 15 07:18:18.079698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:18.079646    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2"
Sep 15 07:18:18.094917 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:18.094883    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r247m"
Sep 15 07:18:18.107125 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:18.107101    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dclwl"
Sep 15 07:18:18.112146 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:18.112125    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zg6jk"
Sep 15 07:18:18.118771 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:18.118750    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zvc79"
Sep 15 07:18:18.126566 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:18.126548    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dbclm"
Sep 15 07:18:18.135630 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:18.135604    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:18.143416 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:18.143391    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lkzcg"
Sep 15 07:18:18.149242 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:18.149221    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4scf4"
Sep 15 07:18:18.394836 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:18.394742    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret\") pod \"global-pull-secret-syncer-kzskh\" (UID: \"08733b0b-6ce9-4bc5-87c5-4565bf0833b5\") " pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:18.395001 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:18.394919    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:18.395072 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:18.395006    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret podName:08733b0b-6ce9-4bc5-87c5-4565bf0833b5 nodeName:}" failed. No retries permitted until 2026-09-15 07:18:19.394983225 +0000 UTC m=+3.941553319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret") pod "global-pull-secret-syncer-kzskh" (UID: "08733b0b-6ce9-4bc5-87c5-4565bf0833b5") : object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:18.495785 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:18.495749    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs\") pod \"network-metrics-daemon-jhh6r\" (UID: \"bf0b2425-be3b-4095-b8ab-a44f426176ac\") " pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:18.495785 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:18.495804    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5kzw\" (UniqueName: \"kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw\") pod \"network-check-target-rfcdm\" (UID: \"7cc100f6-dbfe-412b-9ce2-845dea18988a\") " pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:18.496023 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:18.495921    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:18.496023 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:18.495931    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 07:18:18.496023 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:18.495947    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 07:18:18.496023 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:18.495962    2571 projected.go:196] Error preparing data for projected volume kube-api-access-s5kzw for pod openshift-network-diagnostics/network-check-target-rfcdm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:18.496023 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:18.495998    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs podName:bf0b2425-be3b-4095-b8ab-a44f426176ac nodeName:}" failed. No retries permitted until 2026-09-15 07:18:19.495976341 +0000 UTC m=+4.042546426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs") pod "network-metrics-daemon-jhh6r" (UID: "bf0b2425-be3b-4095-b8ab-a44f426176ac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:18.496023 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:18.496016    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw podName:7cc100f6-dbfe-412b-9ce2-845dea18988a nodeName:}" failed. No retries permitted until 2026-09-15 07:18:19.49600874 +0000 UTC m=+4.042578822 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5kzw" (UniqueName: "kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw") pod "network-check-target-rfcdm" (UID: "7cc100f6-dbfe-412b-9ce2-845dea18988a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:18.812869 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:18.812737    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 07:13:17 +0000 UTC" deadline="2028-06-28 09:11:51.450264532 +0000 UTC"
Sep 15 07:18:18.812869 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:18.812819    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15649h53m32.637458259s"
Sep 15 07:18:19.017811 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:19.015966    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1932db5_d903_4b17_ac25_36613aa04ba4.slice/crio-fa3aa3edc38359f1d71aca42fd06fb4350a8fa72909b84b2bcfa1d04706e4a70 WatchSource:0}: Error finding container fa3aa3edc38359f1d71aca42fd06fb4350a8fa72909b84b2bcfa1d04706e4a70: Status 404 returned error can't find the container with id fa3aa3edc38359f1d71aca42fd06fb4350a8fa72909b84b2bcfa1d04706e4a70
Sep 15 07:18:19.018284 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:19.018241    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5b21b3_5de6_4167_b785_fdf32d77e3af.slice/crio-5180ff35425935271657d7ff475b4e0b55c8686c034eb2b3923d8b74ff34cdb0 WatchSource:0}: Error finding container 5180ff35425935271657d7ff475b4e0b55c8686c034eb2b3923d8b74ff34cdb0: Status 404 returned error can't find the container with id 5180ff35425935271657d7ff475b4e0b55c8686c034eb2b3923d8b74ff34cdb0
Sep 15 07:18:19.021105 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:19.021077    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod850e8885_aca8_4688_8f07_087daf128b3d.slice/crio-bb75377d93ad8bccfdfe44804bcf2788d3973646137e792efb357e97083fbe17 WatchSource:0}: Error finding container bb75377d93ad8bccfdfe44804bcf2788d3973646137e792efb357e97083fbe17: Status 404 returned error can't find the container with id bb75377d93ad8bccfdfe44804bcf2788d3973646137e792efb357e97083fbe17
Sep 15 07:18:19.022543 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:19.022462    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a87606_6562_4269_b2be_0618e3d86409.slice/crio-f7db879c386eaf08428b7ecba6d43595dff84704dcfc20ba7d562982466758a0 WatchSource:0}: Error finding container f7db879c386eaf08428b7ecba6d43595dff84704dcfc20ba7d562982466758a0: Status 404 returned error can't find the container with id f7db879c386eaf08428b7ecba6d43595dff84704dcfc20ba7d562982466758a0
Sep 15 07:18:19.024030 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:19.024006    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecf978c_c2fb_4db3_a60c_d6d4eb3d7f50.slice/crio-0cb195a3cc3e1a0c37ec0110b47b86e2b5dfab51db37abd2bba8275c618a9cdf WatchSource:0}: Error finding container 0cb195a3cc3e1a0c37ec0110b47b86e2b5dfab51db37abd2bba8275c618a9cdf: Status 404 returned error can't find the container with id 0cb195a3cc3e1a0c37ec0110b47b86e2b5dfab51db37abd2bba8275c618a9cdf
Sep 15 07:18:19.027394 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:19.025469    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38076ad_5b5b_4fff_8ad9_149153f9d9d2.slice/crio-281843c9bdbc73b5a07ddf7ea6bd86904e4acd23ca8d46e713c9eb5f10ffed41 WatchSource:0}: Error finding container 281843c9bdbc73b5a07ddf7ea6bd86904e4acd23ca8d46e713c9eb5f10ffed41: Status 404 returned error can't find the container with id 281843c9bdbc73b5a07ddf7ea6bd86904e4acd23ca8d46e713c9eb5f10ffed41
Sep 15 07:18:19.027394 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:19.026467    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061e1b77_3f2d_4f1a_b389_04ffdf5d0530.slice/crio-cf9f31497d8ef28f3438f66f3d157af8ab2e4f8b5edcd14c0477669581a9f63b WatchSource:0}: Error finding container cf9f31497d8ef28f3438f66f3d157af8ab2e4f8b5edcd14c0477669581a9f63b: Status 404 returned error can't find the container with id cf9f31497d8ef28f3438f66f3d157af8ab2e4f8b5edcd14c0477669581a9f63b
Sep 15 07:18:19.027808 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:19.027576    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542e179d_f430_4b8c_83f9_a7b45998ae91.slice/crio-4699d54a8f9f7556758b4febce198700cac799d3b73fa8a5454e5dde0ea727b9 WatchSource:0}: Error finding container 4699d54a8f9f7556758b4febce198700cac799d3b73fa8a5454e5dde0ea727b9: Status 404 returned error can't find the container with id 4699d54a8f9f7556758b4febce198700cac799d3b73fa8a5454e5dde0ea727b9
Sep 15 07:18:19.029732 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:19.028984    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7833b4_e17c_4bc7_bd90_f293e9d38e15.slice/crio-c514c89f6a17defda2e97761c17afc1a917efd5b854a6af806a8dc9d22b635de WatchSource:0}: Error finding container c514c89f6a17defda2e97761c17afc1a917efd5b854a6af806a8dc9d22b635de: Status 404 returned error can't find the container with id c514c89f6a17defda2e97761c17afc1a917efd5b854a6af806a8dc9d22b635de
Sep 15 07:18:19.403060 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:19.402962    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret\") pod \"global-pull-secret-syncer-kzskh\" (UID: \"08733b0b-6ce9-4bc5-87c5-4565bf0833b5\") " pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:19.403234 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:19.403138    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:19.403234 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:19.403223    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret podName:08733b0b-6ce9-4bc5-87c5-4565bf0833b5 nodeName:}" failed. No retries permitted until 2026-09-15 07:18:21.403199635 +0000 UTC m=+5.949769713 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret") pod "global-pull-secret-syncer-kzskh" (UID: "08733b0b-6ce9-4bc5-87c5-4565bf0833b5") : object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:19.504284 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:19.504234    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs\") pod \"network-metrics-daemon-jhh6r\" (UID: \"bf0b2425-be3b-4095-b8ab-a44f426176ac\") " pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:19.504442 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:19.504325    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5kzw\" (UniqueName: \"kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw\") pod \"network-check-target-rfcdm\" (UID: \"7cc100f6-dbfe-412b-9ce2-845dea18988a\") " pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:19.504442 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:19.504392    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:19.504552 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:19.504470    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs podName:bf0b2425-be3b-4095-b8ab-a44f426176ac nodeName:}" failed. No retries permitted until 2026-09-15 07:18:21.504450238 +0000 UTC m=+6.051020344 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs") pod "network-metrics-daemon-jhh6r" (UID: "bf0b2425-be3b-4095-b8ab-a44f426176ac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:19.504552 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:19.504481    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 07:18:19.504552 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:19.504505    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 07:18:19.504552 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:19.504518    2571 projected.go:196] Error preparing data for projected volume kube-api-access-s5kzw for pod openshift-network-diagnostics/network-check-target-rfcdm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:19.504729 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:19.504575    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw podName:7cc100f6-dbfe-412b-9ce2-845dea18988a nodeName:}" failed. No retries permitted until 2026-09-15 07:18:21.504557936 +0000 UTC m=+6.051128024 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5kzw" (UniqueName: "kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw") pod "network-check-target-rfcdm" (UID: "7cc100f6-dbfe-412b-9ce2-845dea18988a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:19.813197 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:19.813061    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 07:13:17 +0000 UTC" deadline="2028-02-10 20:08:53.795686187 +0000 UTC"
Sep 15 07:18:19.813197 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:19.813104    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12324h50m33.982587086s"
Sep 15 07:18:19.910284 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:19.909375    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:19.910284 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:19.909518    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhh6r" podUID="bf0b2425-be3b-4095-b8ab-a44f426176ac"
Sep 15 07:18:19.910284 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:19.909922    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:19.910284 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:19.910019    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzskh" podUID="08733b0b-6ce9-4bc5-87c5-4565bf0833b5"
Sep 15 07:18:19.910284 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:19.910086    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:19.910284 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:19.910163    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfcdm" podUID="7cc100f6-dbfe-412b-9ce2-845dea18988a"
Sep 15 07:18:19.926233 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:19.926205    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.ec2.internal" event={"ID":"ed9aeb0260bd215aa885797c44981ced","Type":"ContainerDied","Data":"8e268e023bb087cec92c0519c6658e35b6449a1a83afc8f6d1fadc3f305ebb9b"}
Sep 15 07:18:19.926404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:19.926381    2571 generic.go:356] "Generic (PLEG): container finished" podID="ed9aeb0260bd215aa885797c44981ced" containerID="8e268e023bb087cec92c0519c6658e35b6449a1a83afc8f6d1fadc3f305ebb9b" exitCode=0
Sep 15 07:18:19.938293 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:19.936147    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.ec2.internal" event={"ID":"457fc68f7214b4c70c98b4a1d69ed2ba","Type":"ContainerStarted","Data":"91ccef339c8dfcf6f7b2d80f5564c1e9a68e43999e11e71a4c3a2176f38b42ad"}
Sep 15 07:18:19.944645 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:19.944612    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r247m" event={"ID":"542e179d-f430-4b8c-83f9-a7b45998ae91","Type":"ContainerStarted","Data":"4699d54a8f9f7556758b4febce198700cac799d3b73fa8a5454e5dde0ea727b9"}
Sep 15 07:18:19.950723 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:19.950688    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4scf4" event={"ID":"c38076ad-5b5b-4fff-8ad9-149153f9d9d2","Type":"ContainerStarted","Data":"281843c9bdbc73b5a07ddf7ea6bd86904e4acd23ca8d46e713c9eb5f10ffed41"}
Sep 15 07:18:19.956785 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:19.956754    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lkzcg" event={"ID":"7ecf978c-c2fb-4db3-a60c-d6d4eb3d7f50","Type":"ContainerStarted","Data":"0cb195a3cc3e1a0c37ec0110b47b86e2b5dfab51db37abd2bba8275c618a9cdf"}
Sep 15 07:18:19.966396 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:19.966369    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zvc79" event={"ID":"a1932db5-d903-4b17-ac25-36613aa04ba4","Type":"ContainerStarted","Data":"fa3aa3edc38359f1d71aca42fd06fb4350a8fa72909b84b2bcfa1d04706e4a70"}
Sep 15 07:18:19.967292 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:19.967241    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.ec2.internal" podStartSLOduration=2.967222916 podStartE2EDuration="2.967222916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:18:19.967100754 +0000 UTC m=+4.513670878" watchObservedRunningTime="2026-09-15 07:18:19.967222916 +0000 UTC m=+4.513793023"
Sep 15 07:18:19.974318 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:19.974252    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zg6jk" event={"ID":"fc7833b4-e17c-4bc7-bd90-f293e9d38e15","Type":"ContainerStarted","Data":"c514c89f6a17defda2e97761c17afc1a917efd5b854a6af806a8dc9d22b635de"}
Sep 15 07:18:19.978798 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:19.978721    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dclwl" event={"ID":"061e1b77-3f2d-4f1a-b389-04ffdf5d0530","Type":"ContainerStarted","Data":"cf9f31497d8ef28f3438f66f3d157af8ab2e4f8b5edcd14c0477669581a9f63b"}
Sep 15 07:18:19.983104 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:19.983080    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dbclm" event={"ID":"850e8885-aca8-4688-8f07-087daf128b3d","Type":"ContainerStarted","Data":"bb75377d93ad8bccfdfe44804bcf2788d3973646137e792efb357e97083fbe17"}
Sep 15 07:18:19.987893 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:19.987867    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtrll" event={"ID":"b5a87606-6562-4269-b2be-0618e3d86409","Type":"ContainerStarted","Data":"f7db879c386eaf08428b7ecba6d43595dff84704dcfc20ba7d562982466758a0"}
Sep 15 07:18:19.989492 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:19.989468    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2" event={"ID":"8c5b21b3-5de6-4167-b785-fdf32d77e3af","Type":"ContainerStarted","Data":"5180ff35425935271657d7ff475b4e0b55c8686c034eb2b3923d8b74ff34cdb0"}
Sep 15 07:18:21.030905 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:21.030836    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.ec2.internal" event={"ID":"ed9aeb0260bd215aa885797c44981ced","Type":"ContainerStarted","Data":"ef4b42642cf280227744106183671a0679f0ade91458b706c8b372ba03312260"}
Sep 15 07:18:21.421343 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:21.420472    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret\") pod \"global-pull-secret-syncer-kzskh\" (UID: \"08733b0b-6ce9-4bc5-87c5-4565bf0833b5\") " pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:21.421343 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:21.420662    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:21.421343 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:21.420728    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret podName:08733b0b-6ce9-4bc5-87c5-4565bf0833b5 nodeName:}" failed. No retries permitted until 2026-09-15 07:18:25.420709723 +0000 UTC m=+9.967279801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret") pod "global-pull-secret-syncer-kzskh" (UID: "08733b0b-6ce9-4bc5-87c5-4565bf0833b5") : object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:21.521137 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:21.521095    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5kzw\" (UniqueName: \"kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw\") pod \"network-check-target-rfcdm\" (UID: \"7cc100f6-dbfe-412b-9ce2-845dea18988a\") " pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:21.521359 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:21.521198    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs\") pod \"network-metrics-daemon-jhh6r\" (UID: \"bf0b2425-be3b-4095-b8ab-a44f426176ac\") " pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:21.521359 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:21.521330    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:21.521515 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:21.521397    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs podName:bf0b2425-be3b-4095-b8ab-a44f426176ac nodeName:}" failed. No retries permitted until 2026-09-15 07:18:25.521378811 +0000 UTC m=+10.067948902 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs") pod "network-metrics-daemon-jhh6r" (UID: "bf0b2425-be3b-4095-b8ab-a44f426176ac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:21.521833 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:21.521811    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 07:18:21.521925 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:21.521841    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 07:18:21.521925 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:21.521855    2571 projected.go:196] Error preparing data for projected volume kube-api-access-s5kzw for pod openshift-network-diagnostics/network-check-target-rfcdm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:21.521925 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:21.521900    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw podName:7cc100f6-dbfe-412b-9ce2-845dea18988a nodeName:}" failed. No retries permitted until 2026-09-15 07:18:25.521885623 +0000 UTC m=+10.068455714 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5kzw" (UniqueName: "kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw") pod "network-check-target-rfcdm" (UID: "7cc100f6-dbfe-412b-9ce2-845dea18988a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:21.902585 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:21.902506    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:21.902807 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:21.902654    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhh6r" podUID="bf0b2425-be3b-4095-b8ab-a44f426176ac"
Sep 15 07:18:21.903113 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:21.903064    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:21.903185 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:21.903157    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzskh" podUID="08733b0b-6ce9-4bc5-87c5-4565bf0833b5"
Sep 15 07:18:21.903244 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:21.903210    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:21.903462 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:21.903436    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfcdm" podUID="7cc100f6-dbfe-412b-9ce2-845dea18988a"
Sep 15 07:18:23.906912 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:23.906882    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:23.907437 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:23.907013    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhh6r" podUID="bf0b2425-be3b-4095-b8ab-a44f426176ac"
Sep 15 07:18:23.907437 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:23.907389    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:23.907540 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:23.907490    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzskh" podUID="08733b0b-6ce9-4bc5-87c5-4565bf0833b5"
Sep 15 07:18:23.907540 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:23.907532    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:23.907633 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:23.907598    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfcdm" podUID="7cc100f6-dbfe-412b-9ce2-845dea18988a"
Sep 15 07:18:25.459048 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:25.459007    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret\") pod \"global-pull-secret-syncer-kzskh\" (UID: \"08733b0b-6ce9-4bc5-87c5-4565bf0833b5\") " pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:25.459515 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:25.459202    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:25.459515 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:25.459279    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret podName:08733b0b-6ce9-4bc5-87c5-4565bf0833b5 nodeName:}" failed. No retries permitted until 2026-09-15 07:18:33.459246675 +0000 UTC m=+18.005816766 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret") pod "global-pull-secret-syncer-kzskh" (UID: "08733b0b-6ce9-4bc5-87c5-4565bf0833b5") : object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:25.559676 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:25.559432    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5kzw\" (UniqueName: \"kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw\") pod \"network-check-target-rfcdm\" (UID: \"7cc100f6-dbfe-412b-9ce2-845dea18988a\") " pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:25.559676 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:25.559531    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs\") pod \"network-metrics-daemon-jhh6r\" (UID: \"bf0b2425-be3b-4095-b8ab-a44f426176ac\") " pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:25.559676 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:25.559578    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 07:18:25.559676 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:25.559602    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 07:18:25.559676 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:25.559615    2571 projected.go:196] Error preparing data for projected volume kube-api-access-s5kzw for pod openshift-network-diagnostics/network-check-target-rfcdm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:25.559676 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:25.559621    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:25.559676 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:25.559668    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw podName:7cc100f6-dbfe-412b-9ce2-845dea18988a nodeName:}" failed. No retries permitted until 2026-09-15 07:18:33.559651281 +0000 UTC m=+18.106221373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5kzw" (UniqueName: "kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw") pod "network-check-target-rfcdm" (UID: "7cc100f6-dbfe-412b-9ce2-845dea18988a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:25.559676 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:25.559686    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs podName:bf0b2425-be3b-4095-b8ab-a44f426176ac nodeName:}" failed. No retries permitted until 2026-09-15 07:18:33.559678359 +0000 UTC m=+18.106248439 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs") pod "network-metrics-daemon-jhh6r" (UID: "bf0b2425-be3b-4095-b8ab-a44f426176ac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:25.904174 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:25.904087    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:25.904345 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:25.904189    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:25.904345 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:25.904198    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzskh" podUID="08733b0b-6ce9-4bc5-87c5-4565bf0833b5"
Sep 15 07:18:25.904345 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:25.904315    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfcdm" podUID="7cc100f6-dbfe-412b-9ce2-845dea18988a"
Sep 15 07:18:25.904510 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:25.904382    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:25.904510 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:25.904463    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhh6r" podUID="bf0b2425-be3b-4095-b8ab-a44f426176ac"
Sep 15 07:18:27.903833 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:27.903348    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:27.903833 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:27.903485    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhh6r" podUID="bf0b2425-be3b-4095-b8ab-a44f426176ac"
Sep 15 07:18:27.903833 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:27.903523    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:27.903833 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:27.903646    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzskh" podUID="08733b0b-6ce9-4bc5-87c5-4565bf0833b5"
Sep 15 07:18:27.903833 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:27.903695    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:27.903833 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:27.903760    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfcdm" podUID="7cc100f6-dbfe-412b-9ce2-845dea18988a"
Sep 15 07:18:29.903037 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:29.902960    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:29.903493 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:29.902961    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:29.903493 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:29.903080    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzskh" podUID="08733b0b-6ce9-4bc5-87c5-4565bf0833b5"
Sep 15 07:18:29.903493 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:29.903105    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:29.903493 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:29.903191    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfcdm" podUID="7cc100f6-dbfe-412b-9ce2-845dea18988a"
Sep 15 07:18:29.903493 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:29.903311    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhh6r" podUID="bf0b2425-be3b-4095-b8ab-a44f426176ac"
Sep 15 07:18:31.902871 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:31.902824    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:31.903330 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:31.902881    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:31.903330 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:31.902824    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:31.903330 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:31.902965    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfcdm" podUID="7cc100f6-dbfe-412b-9ce2-845dea18988a"
Sep 15 07:18:31.903330 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:31.903058    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhh6r" podUID="bf0b2425-be3b-4095-b8ab-a44f426176ac"
Sep 15 07:18:31.903330 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:31.903132    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzskh" podUID="08733b0b-6ce9-4bc5-87c5-4565bf0833b5"
Sep 15 07:18:33.518973 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:33.518782    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret\") pod \"global-pull-secret-syncer-kzskh\" (UID: \"08733b0b-6ce9-4bc5-87c5-4565bf0833b5\") " pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:33.519478 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:33.518937    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:33.519478 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:33.519098    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret podName:08733b0b-6ce9-4bc5-87c5-4565bf0833b5 nodeName:}" failed. No retries permitted until 2026-09-15 07:18:49.519077666 +0000 UTC m=+34.065647762 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret") pod "global-pull-secret-syncer-kzskh" (UID: "08733b0b-6ce9-4bc5-87c5-4565bf0833b5") : object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:33.620071 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:33.620037    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs\") pod \"network-metrics-daemon-jhh6r\" (UID: \"bf0b2425-be3b-4095-b8ab-a44f426176ac\") " pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:33.620220 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:33.620088    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5kzw\" (UniqueName: \"kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw\") pod \"network-check-target-rfcdm\" (UID: \"7cc100f6-dbfe-412b-9ce2-845dea18988a\") " pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:33.620316 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:33.620210    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:33.620316 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:33.620238    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 07:18:33.620316 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:33.620271    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 07:18:33.620316 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:33.620285    2571 projected.go:196] Error preparing data for projected volume kube-api-access-s5kzw for pod openshift-network-diagnostics/network-check-target-rfcdm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:33.620316 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:33.620310    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs podName:bf0b2425-be3b-4095-b8ab-a44f426176ac nodeName:}" failed. No retries permitted until 2026-09-15 07:18:49.620287988 +0000 UTC m=+34.166858068 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs") pod "network-metrics-daemon-jhh6r" (UID: "bf0b2425-be3b-4095-b8ab-a44f426176ac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:33.620507 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:33.620331    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw podName:7cc100f6-dbfe-412b-9ce2-845dea18988a nodeName:}" failed. No retries permitted until 2026-09-15 07:18:49.620319741 +0000 UTC m=+34.166889839 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5kzw" (UniqueName: "kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw") pod "network-check-target-rfcdm" (UID: "7cc100f6-dbfe-412b-9ce2-845dea18988a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:33.902818 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:33.902701    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:33.902818 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:33.902702    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:33.903031 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:33.902834    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:33.903031 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:33.902823    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfcdm" podUID="7cc100f6-dbfe-412b-9ce2-845dea18988a"
Sep 15 07:18:33.903031 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:33.902925    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhh6r" podUID="bf0b2425-be3b-4095-b8ab-a44f426176ac"
Sep 15 07:18:33.903031 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:33.902993    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzskh" podUID="08733b0b-6ce9-4bc5-87c5-4565bf0833b5"
Sep 15 07:18:35.903441 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:35.903405    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:35.903829 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:35.903495    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzskh" podUID="08733b0b-6ce9-4bc5-87c5-4565bf0833b5"
Sep 15 07:18:35.903829 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:35.903508    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:35.903829 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:35.903543    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:35.903829 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:35.903621    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfcdm" podUID="7cc100f6-dbfe-412b-9ce2-845dea18988a"
Sep 15 07:18:35.903829 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:35.903698    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhh6r" podUID="bf0b2425-be3b-4095-b8ab-a44f426176ac"
Sep 15 07:18:37.074794 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:37.074096    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dclwl" event={"ID":"061e1b77-3f2d-4f1a-b389-04ffdf5d0530","Type":"ContainerStarted","Data":"1629f61f420c9c2ad435c832b9f9e9aeb494d5747611f5946dbbbacb62dd9b9d"}
Sep 15 07:18:37.076928 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:37.076893    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dbclm" event={"ID":"850e8885-aca8-4688-8f07-087daf128b3d","Type":"ContainerStarted","Data":"20848fa78a757b37eb96c9caba65c34422bafc639547675a3c112abba81bac7c"}
Sep 15 07:18:37.079144 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:37.079119    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtrll" event={"ID":"b5a87606-6562-4269-b2be-0618e3d86409","Type":"ContainerStarted","Data":"92b97bf016f193732cfca69ade131ab3c981d931cab51d922f12459365d1c3a1"}
Sep 15 07:18:37.079218 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:37.079195    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtrll" event={"ID":"b5a87606-6562-4269-b2be-0618e3d86409","Type":"ContainerStarted","Data":"80a72ca01727f814b954493569172e9ef87561541455acfc0547da69b5813c1e"}
Sep 15 07:18:37.080755 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:37.080733    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2" event={"ID":"8c5b21b3-5de6-4167-b785-fdf32d77e3af","Type":"ContainerStarted","Data":"85dd63f23690dcdc3fb9baf914cce70a4bf7250b3947cd9b48ee12790d1503f5"}
Sep 15 07:18:37.082302 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:37.082277    2571 generic.go:356] "Generic (PLEG): container finished" podID="542e179d-f430-4b8c-83f9-a7b45998ae91" containerID="8bc1fb985a96f0193f14fba498111e4bec836e8111c64fd92fc3d94163c23d55" exitCode=0
Sep 15 07:18:37.082409 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:37.082340    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r247m" event={"ID":"542e179d-f430-4b8c-83f9-a7b45998ae91","Type":"ContainerDied","Data":"8bc1fb985a96f0193f14fba498111e4bec836e8111c64fd92fc3d94163c23d55"}
Sep 15 07:18:37.084624 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:37.084598    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4scf4" event={"ID":"c38076ad-5b5b-4fff-8ad9-149153f9d9d2","Type":"ContainerStarted","Data":"be661a46e70d5618a65837f5de409c5b91887e445d9c9816956bdac4666b2173"}
Sep 15 07:18:37.086303 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:37.086277    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zvc79" event={"ID":"a1932db5-d903-4b17-ac25-36613aa04ba4","Type":"ContainerStarted","Data":"12f7ddd98781f364930ba4f05594463efe056219dd6bacc76494e07d73ad736a"}
Sep 15 07:18:37.087718 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:37.087698    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zg6jk" event={"ID":"fc7833b4-e17c-4bc7-bd90-f293e9d38e15","Type":"ContainerStarted","Data":"3d3ddc9773c14354e46c2cbb1af73cda70ea264daf3057fc6d0f6b4f44e90d13"}
Sep 15 07:18:37.088909 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:37.088861    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.ec2.internal" podStartSLOduration=20.088845121 podStartE2EDuration="20.088845121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:18:21.047472623 +0000 UTC m=+5.594042723" watchObservedRunningTime="2026-09-15 07:18:37.088845121 +0000 UTC m=+21.635415220"
Sep 15 07:18:37.089428 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:37.089391    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dclwl" podStartSLOduration=3.414306311 podStartE2EDuration="21.089381882s" totalImagesPullingTime="17.675075571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:18:37.088526698 +0000 UTC m=+21.635096796" watchObservedRunningTime="2026-09-15 07:18:37.089381882 +0000 UTC m=+21.635951986"
Sep 15 07:18:37.103185 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:37.103134    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zvc79" podStartSLOduration=3.424398201 podStartE2EDuration="21.103113167s" totalImagesPullingTime="17.678714966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:18:37.102914714 +0000 UTC m=+21.649484834" watchObservedRunningTime="2026-09-15 07:18:37.103113167 +0000 UTC m=+21.649683268"
Sep 15 07:18:37.119716 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:37.119672    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zg6jk" podStartSLOduration=3.431930071 podStartE2EDuration="21.119656357s" totalImagesPullingTime="17.687726286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:18:37.119111355 +0000 UTC m=+21.665681454" watchObservedRunningTime="2026-09-15 07:18:37.119656357 +0000 UTC m=+21.666226456"
Sep 15 07:18:37.135588 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:37.135535    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dbclm" podStartSLOduration=3.403571731 podStartE2EDuration="21.135519995s" totalImagesPullingTime="17.731948264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:18:37.135174297 +0000 UTC m=+21.681744398" watchObservedRunningTime="2026-09-15 07:18:37.135519995 +0000 UTC m=+21.682090094"
Sep 15 07:18:37.181577 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:37.181536    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4scf4" podStartSLOduration=3.508689313 podStartE2EDuration="21.1815205s" totalImagesPullingTime="17.672831187s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:18:37.158382809 +0000 UTC m=+21.704952908" watchObservedRunningTime="2026-09-15 07:18:37.1815205 +0000 UTC m=+21.728090600"
Sep 15 07:18:37.902728 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:37.902702    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:37.902878 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:37.902705    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:37.902878 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:37.902840    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfcdm" podUID="7cc100f6-dbfe-412b-9ce2-845dea18988a"
Sep 15 07:18:37.902994 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:37.902706    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:37.902994 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:37.902919    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzskh" podUID="08733b0b-6ce9-4bc5-87c5-4565bf0833b5"
Sep 15 07:18:37.903070 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:37.903040    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhh6r" podUID="bf0b2425-be3b-4095-b8ab-a44f426176ac"
Sep 15 07:18:38.094197 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:38.094111    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtrll" event={"ID":"b5a87606-6562-4269-b2be-0618e3d86409","Type":"ContainerStarted","Data":"f14599a26dd391516ae857111fa9345e4cfde2d8bcf172c0453d272e4cd5ebf5"}
Sep 15 07:18:38.094977 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:38.094199    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtrll" event={"ID":"b5a87606-6562-4269-b2be-0618e3d86409","Type":"ContainerStarted","Data":"cdaf0dbd4c81774a2a7c0c131c4396d65eb6e75843a84bbfb3645c0c9ca6f354"}
Sep 15 07:18:38.094977 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:38.094235    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtrll" event={"ID":"b5a87606-6562-4269-b2be-0618e3d86409","Type":"ContainerStarted","Data":"4be75504d5401212ab01d5fc9b1ad73703eaceb42f5eed0db57aca9bd3832e5b"}
Sep 15 07:18:38.094977 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:38.094288    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtrll" event={"ID":"b5a87606-6562-4269-b2be-0618e3d86409","Type":"ContainerStarted","Data":"4d8249f860fd38060372afd8018ac6a5007d1271bd327ecbf3c5fe32a5952c2d"}
Sep 15 07:18:38.096328 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:38.095952    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lkzcg" event={"ID":"7ecf978c-c2fb-4db3-a60c-d6d4eb3d7f50","Type":"ContainerStarted","Data":"fe874ea7cb69c86ffe03a94af8f0b774a23fcb08852b035da75041105a54ac58"}
Sep 15 07:18:38.110965 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:38.110922    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lkzcg" podStartSLOduration=4.439180604 podStartE2EDuration="22.110910889s" totalImagesPullingTime="17.671730285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:18:38.110817401 +0000 UTC m=+22.657387501" watchObservedRunningTime="2026-09-15 07:18:38.110910889 +0000 UTC m=+22.657480989"
Sep 15 07:18:38.375727 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:38.375562    2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 07:18:38.830328 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:38.830176    2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T07:18:38.37572093Z","UUID":"2103b1ca-38b8-4cf5-845d-104d52ccf15e","Handler":null,"Name":"","Endpoint":""}
Sep 15 07:18:38.831599 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:38.831581    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 07:18:38.831698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:38.831611    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 07:18:39.099988 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:39.099909    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2" event={"ID":"8c5b21b3-5de6-4167-b785-fdf32d77e3af","Type":"ContainerStarted","Data":"2385a34e1b42ebef47b8e00409bf8cda56af2a86316d46d0c5867d926e16c6f8"}
Sep 15 07:18:39.902927 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:39.902895    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:39.903090 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:39.902897    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:39.903090 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:39.903010    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzskh" podUID="08733b0b-6ce9-4bc5-87c5-4565bf0833b5"
Sep 15 07:18:39.903090 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:39.902897    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:39.903090 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:39.903070    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfcdm" podUID="7cc100f6-dbfe-412b-9ce2-845dea18988a"
Sep 15 07:18:39.903238 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:39.903135    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhh6r" podUID="bf0b2425-be3b-4095-b8ab-a44f426176ac"
Sep 15 07:18:40.106589 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:40.106557    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtrll" event={"ID":"b5a87606-6562-4269-b2be-0618e3d86409","Type":"ContainerStarted","Data":"595b9a5bd38bb9d6d9861d47726bf1ea29ba7952db5a067c761757a2238a65fe"}
Sep 15 07:18:41.111581 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:41.111543    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2" event={"ID":"8c5b21b3-5de6-4167-b785-fdf32d77e3af","Type":"ContainerStarted","Data":"b1e5e610da5f49bfbf27aff26442ac12e7434c9eb5cf8d2996afba6d67d5ba81"}
Sep 15 07:18:41.133488 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:41.133429    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdqf2" podStartSLOduration=4.302229105 podStartE2EDuration="25.133410912s" totalImagesPullingTime="20.831181807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:18:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:18:41.133062079 +0000 UTC m=+25.679632200" watchObservedRunningTime="2026-09-15 07:18:41.133410912 +0000 UTC m=+25.679981011"
Sep 15 07:18:41.751596 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:41.751561    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dclwl"
Sep 15 07:18:41.752195 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:41.752171    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dclwl"
Sep 15 07:18:41.902853 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:41.902771    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:41.903022 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:41.902771    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:41.903022 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:41.902900    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfcdm" podUID="7cc100f6-dbfe-412b-9ce2-845dea18988a"
Sep 15 07:18:41.903022 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:41.902978    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzskh" podUID="08733b0b-6ce9-4bc5-87c5-4565bf0833b5"
Sep 15 07:18:41.903022 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:41.902772    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:41.903198 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:41.903079    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhh6r" podUID="bf0b2425-be3b-4095-b8ab-a44f426176ac"
Sep 15 07:18:42.114092 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:42.114056    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dclwl"
Sep 15 07:18:42.114508 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:42.114351    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dclwl"
Sep 15 07:18:43.119218 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:43.118967    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtrll" event={"ID":"b5a87606-6562-4269-b2be-0618e3d86409","Type":"ContainerStarted","Data":"9a1727814dda82d3445b62bd0bd749b24a1ca6cc3959a3dd994b52daf0d3c040"}
Sep 15 07:18:43.119827 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:43.119291    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:43.120847 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:43.120826    2571 generic.go:356] "Generic (PLEG): container finished" podID="542e179d-f430-4b8c-83f9-a7b45998ae91" containerID="fd091c472cba2be4d88e684ff3581de8127661fb5bbc93e56bb63e62da32e39d" exitCode=0
Sep 15 07:18:43.120973 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:43.120909    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r247m" event={"ID":"542e179d-f430-4b8c-83f9-a7b45998ae91","Type":"ContainerDied","Data":"fd091c472cba2be4d88e684ff3581de8127661fb5bbc93e56bb63e62da32e39d"}
Sep 15 07:18:43.134864 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:43.134843    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:43.145033 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:43.144996    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qtrll" podStartSLOduration=9.411896761 podStartE2EDuration="27.14498155s" totalImagesPullingTime="17.733084789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:18:43.14347309 +0000 UTC m=+27.690043212" watchObservedRunningTime="2026-09-15 07:18:43.14498155 +0000 UTC m=+27.691551696"
Sep 15 07:18:43.903302 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:43.903267    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:43.903302 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:43.903291    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:43.903558 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:43.903330    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:43.903558 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:43.903419    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzskh" podUID="08733b0b-6ce9-4bc5-87c5-4565bf0833b5"
Sep 15 07:18:43.903631 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:43.903566    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhh6r" podUID="bf0b2425-be3b-4095-b8ab-a44f426176ac"
Sep 15 07:18:43.903675 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:43.903649    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfcdm" podUID="7cc100f6-dbfe-412b-9ce2-845dea18988a"
Sep 15 07:18:44.124203 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:44.124174    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:44.124830 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:44.124265    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:44.140527 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:44.140500    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:18:44.583163 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:44.583072    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rfcdm"]
Sep 15 07:18:44.583288 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:44.583195    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:44.583353 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:44.583319    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfcdm" podUID="7cc100f6-dbfe-412b-9ce2-845dea18988a"
Sep 15 07:18:44.586402 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:44.586380    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kzskh"]
Sep 15 07:18:44.586501 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:44.586465    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:44.586558 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:44.586541    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzskh" podUID="08733b0b-6ce9-4bc5-87c5-4565bf0833b5"
Sep 15 07:18:44.587147 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:44.587123    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jhh6r"]
Sep 15 07:18:44.587238 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:44.587217    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:44.587368 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:44.587343    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhh6r" podUID="bf0b2425-be3b-4095-b8ab-a44f426176ac"
Sep 15 07:18:45.127485 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:45.127452    2571 generic.go:356] "Generic (PLEG): container finished" podID="542e179d-f430-4b8c-83f9-a7b45998ae91" containerID="7aa20a9873fd026056d201068445bd38b54b4a4bbdbf1719c40e88ca4b934496" exitCode=0
Sep 15 07:18:45.127908 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:45.127538    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r247m" event={"ID":"542e179d-f430-4b8c-83f9-a7b45998ae91","Type":"ContainerDied","Data":"7aa20a9873fd026056d201068445bd38b54b4a4bbdbf1719c40e88ca4b934496"}
Sep 15 07:18:45.905020 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:45.904779    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:45.905158 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:45.904837    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:45.905158 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:45.905123    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzskh" podUID="08733b0b-6ce9-4bc5-87c5-4565bf0833b5"
Sep 15 07:18:45.905238 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:45.905169    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhh6r" podUID="bf0b2425-be3b-4095-b8ab-a44f426176ac"
Sep 15 07:18:46.903032 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:46.902971    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:46.903379 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:46.903064    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfcdm" podUID="7cc100f6-dbfe-412b-9ce2-845dea18988a"
Sep 15 07:18:47.134340 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:47.134306    2571 generic.go:356] "Generic (PLEG): container finished" podID="542e179d-f430-4b8c-83f9-a7b45998ae91" containerID="741850a3bc69e774f0fc7614181f23b23cadd21805e0978260ca54a6fe6130a3" exitCode=0
Sep 15 07:18:47.134340 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:47.134340    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r247m" event={"ID":"542e179d-f430-4b8c-83f9-a7b45998ae91","Type":"ContainerDied","Data":"741850a3bc69e774f0fc7614181f23b23cadd21805e0978260ca54a6fe6130a3"}
Sep 15 07:18:47.903498 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:47.903469    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:47.903848 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:47.903470    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:47.903848 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:47.903581    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhh6r" podUID="bf0b2425-be3b-4095-b8ab-a44f426176ac"
Sep 15 07:18:47.903848 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:47.903682    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzskh" podUID="08733b0b-6ce9-4bc5-87c5-4565bf0833b5"
Sep 15 07:18:48.903412 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:48.903381    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:48.903573 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:48.903482    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfcdm" podUID="7cc100f6-dbfe-412b-9ce2-845dea18988a"
Sep 15 07:18:49.537812 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.537721    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret\") pod \"global-pull-secret-syncer-kzskh\" (UID: \"08733b0b-6ce9-4bc5-87c5-4565bf0833b5\") " pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:49.538013 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:49.537886    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:49.538013 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:49.537956    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret podName:08733b0b-6ce9-4bc5-87c5-4565bf0833b5 nodeName:}" failed. No retries permitted until 2026-09-15 07:19:21.537940072 +0000 UTC m=+66.084510151 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret") pod "global-pull-secret-syncer-kzskh" (UID: "08733b0b-6ce9-4bc5-87c5-4565bf0833b5") : object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:49.638963 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.638927    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs\") pod \"network-metrics-daemon-jhh6r\" (UID: \"bf0b2425-be3b-4095-b8ab-a44f426176ac\") " pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:49.639136 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.638981    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5kzw\" (UniqueName: \"kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw\") pod \"network-check-target-rfcdm\" (UID: \"7cc100f6-dbfe-412b-9ce2-845dea18988a\") " pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:49.639136 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:49.639103    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 07:18:49.639136 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:49.639102    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:49.639136 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:49.639118    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 07:18:49.639136 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:49.639130    2571 projected.go:196] Error preparing data for projected volume kube-api-access-s5kzw for pod openshift-network-diagnostics/network-check-target-rfcdm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:49.639345 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:49.639175    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw podName:7cc100f6-dbfe-412b-9ce2-845dea18988a nodeName:}" failed. No retries permitted until 2026-09-15 07:19:21.639160901 +0000 UTC m=+66.185730978 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5kzw" (UniqueName: "kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw") pod "network-check-target-rfcdm" (UID: "7cc100f6-dbfe-412b-9ce2-845dea18988a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:49.639345 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:49.639194    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs podName:bf0b2425-be3b-4095-b8ab-a44f426176ac nodeName:}" failed. No retries permitted until 2026-09-15 07:19:21.639184555 +0000 UTC m=+66.185754633 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs") pod "network-metrics-daemon-jhh6r" (UID: "bf0b2425-be3b-4095-b8ab-a44f426176ac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:49.707268 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.707225    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-76.ec2.internal" event="NodeReady"
Sep 15 07:18:49.707415 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.707363    2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 07:18:49.744515 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.744488    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x"]
Sep 15 07:18:49.748130 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.748107    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5bd85df4cd-n9nj4"]
Sep 15 07:18:49.748247 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.748236    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x"
Sep 15 07:18:49.750523 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.750500    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 15 07:18:49.750657 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.750635    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nsfj7\""
Sep 15 07:18:49.750769 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.750752    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 15 07:18:49.751858 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.751842    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.756552 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.756537    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 07:18:49.757425 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.757405    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rxml2\""
Sep 15 07:18:49.757498 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.757479    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 07:18:49.757732 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.757720    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 07:18:49.762413 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.762327    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 07:18:49.762793 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.762769    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x"]
Sep 15 07:18:49.767476 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.767437    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qvk9j"]
Sep 15 07:18:49.772153 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.772127    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bd85df4cd-n9nj4"]
Sep 15 07:18:49.772292 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.772175    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5ltrc"]
Sep 15 07:18:49.773952 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.773445    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:18:49.776535 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.776513    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 07:18:49.776660 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.776647    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5ltrc"
Sep 15 07:18:49.776782 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.776765    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tfj9f\""
Sep 15 07:18:49.776892 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.776767    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 07:18:49.778810 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.778791    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 07:18:49.779111 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.779093    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 07:18:49.779280 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.779248    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 07:18:49.779384 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.779327    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fpxmd\""
Sep 15 07:18:49.788041 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.787985    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qvk9j"]
Sep 15 07:18:49.788771 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.788745    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5ltrc"]
Sep 15 07:18:49.840450 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.840418    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b8697c4f-de34-4c3f-ae67-fbbb429010c0-ca-trust-extracted\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.840641 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.840482    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmrrh\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-kube-api-access-wmrrh\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.840641 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.840541    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-bound-sa-token\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.840641 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.840574    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6a8acf07-ffd3-4101-a1d5-16770855954c-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-wdl2x\" (UID: \"6a8acf07-ffd3-4101-a1d5-16770855954c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x"
Sep 15 07:18:49.840641 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.840599    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-wdl2x\" (UID: \"6a8acf07-ffd3-4101-a1d5-16770855954c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x"
Sep 15 07:18:49.840641 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.840623    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.840641 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.840639    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-certificates\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.840951 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.840652    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b8697c4f-de34-4c3f-ae67-fbbb429010c0-trusted-ca\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.840951 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.840714    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b8697c4f-de34-4c3f-ae67-fbbb429010c0-image-registry-private-configuration\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.840951 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.840755    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b8697c4f-de34-4c3f-ae67-fbbb429010c0-installation-pull-secrets\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.903171 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.903140    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:18:49.903398 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.903181    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:18:49.906652 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.906628    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 07:18:49.907030 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.906925    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ftj64\""
Sep 15 07:18:49.907030 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.906926    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 07:18:49.910865 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.910819    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dclwl_061e1b77-3f2d-4f1a-b389-04ffdf5d0530/konnectivity-agent/0.log" limit=1024
Sep 15 07:18:49.941617 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.941585    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmrrh\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-kube-api-access-wmrrh\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.941617 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.941620    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1934978a-de53-4efa-8cb8-2f2bc312ceaf-cert\") pod \"ingress-canary-5ltrc\" (UID: \"1934978a-de53-4efa-8cb8-2f2bc312ceaf\") " pod="openshift-ingress-canary/ingress-canary-5ltrc"
Sep 15 07:18:49.941843 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.941653    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46hq6\" (UniqueName: \"kubernetes.io/projected/64a7e365-d705-4174-992f-bd271f4b418e-kube-api-access-46hq6\") pod \"dns-default-qvk9j\" (UID: \"64a7e365-d705-4174-992f-bd271f4b418e\") " pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:18:49.941843 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.941678    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fksm\" (UniqueName: \"kubernetes.io/projected/1934978a-de53-4efa-8cb8-2f2bc312ceaf-kube-api-access-6fksm\") pod \"ingress-canary-5ltrc\" (UID: \"1934978a-de53-4efa-8cb8-2f2bc312ceaf\") " pod="openshift-ingress-canary/ingress-canary-5ltrc"
Sep 15 07:18:49.941843 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.941772    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-bound-sa-token\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.941843 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.941808    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/64a7e365-d705-4174-992f-bd271f4b418e-config-volume\") pod \"dns-default-qvk9j\" (UID: \"64a7e365-d705-4174-992f-bd271f4b418e\") " pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:18:49.941843 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.941828    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls\") pod \"dns-default-qvk9j\" (UID: \"64a7e365-d705-4174-992f-bd271f4b418e\") " pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:18:49.942067 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.941898    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6a8acf07-ffd3-4101-a1d5-16770855954c-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-wdl2x\" (UID: \"6a8acf07-ffd3-4101-a1d5-16770855954c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x"
Sep 15 07:18:49.942067 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.941934    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-wdl2x\" (UID: \"6a8acf07-ffd3-4101-a1d5-16770855954c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x"
Sep 15 07:18:49.942067 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.941965    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.942067 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.941990    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-certificates\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.942067 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.942013    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b8697c4f-de34-4c3f-ae67-fbbb429010c0-trusted-ca\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.942067 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:49.942048    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 07:18:49.942359 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:49.942086    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 07:18:49.942359 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:49.942098    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bd85df4cd-n9nj4: secret "image-registry-tls" not found
Sep 15 07:18:49.942359 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:49.942113    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert podName:6a8acf07-ffd3-4101-a1d5-16770855954c nodeName:}" failed. No retries permitted until 2026-09-15 07:18:50.442091938 +0000 UTC m=+34.988662030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-wdl2x" (UID: "6a8acf07-ffd3-4101-a1d5-16770855954c") : secret "networking-console-plugin-cert" not found
Sep 15 07:18:49.942359 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:49.942143    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls podName:b8697c4f-de34-4c3f-ae67-fbbb429010c0 nodeName:}" failed. No retries permitted until 2026-09-15 07:18:50.442129897 +0000 UTC m=+34.988699974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls") pod "image-registry-5bd85df4cd-n9nj4" (UID: "b8697c4f-de34-4c3f-ae67-fbbb429010c0") : secret "image-registry-tls" not found
Sep 15 07:18:49.942359 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.942183    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b8697c4f-de34-4c3f-ae67-fbbb429010c0-image-registry-private-configuration\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.942359 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.942212    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b8697c4f-de34-4c3f-ae67-fbbb429010c0-installation-pull-secrets\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.942359 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.942239    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b8697c4f-de34-4c3f-ae67-fbbb429010c0-ca-trust-extracted\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.942359 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.942277    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64a7e365-d705-4174-992f-bd271f4b418e-tmp-dir\") pod \"dns-default-qvk9j\" (UID: \"64a7e365-d705-4174-992f-bd271f4b418e\") " pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:18:49.943094 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.942842    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6a8acf07-ffd3-4101-a1d5-16770855954c-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-wdl2x\" (UID: \"6a8acf07-ffd3-4101-a1d5-16770855954c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x"
Sep 15 07:18:49.943094 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.943056    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b8697c4f-de34-4c3f-ae67-fbbb429010c0-ca-trust-extracted\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.943230 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.943125    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b8697c4f-de34-4c3f-ae67-fbbb429010c0-trusted-ca\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.943297 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.943277    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-certificates\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.946602 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.946567    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b8697c4f-de34-4c3f-ae67-fbbb429010c0-installation-pull-secrets\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.946602 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.946587    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b8697c4f-de34-4c3f-ae67-fbbb429010c0-image-registry-private-configuration\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.951230 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.951209    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-bound-sa-token\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:49.951343 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:49.951292    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmrrh\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-kube-api-access-wmrrh\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:50.018688 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:50.018646    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dclwl_061e1b77-3f2d-4f1a-b389-04ffdf5d0530/konnectivity-agent/0.log" limit=1024
Sep 15 07:18:50.043581 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:50.043508    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64a7e365-d705-4174-992f-bd271f4b418e-tmp-dir\") pod \"dns-default-qvk9j\" (UID: \"64a7e365-d705-4174-992f-bd271f4b418e\") " pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:18:50.043581 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:50.043541    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1934978a-de53-4efa-8cb8-2f2bc312ceaf-cert\") pod \"ingress-canary-5ltrc\" (UID: \"1934978a-de53-4efa-8cb8-2f2bc312ceaf\") " pod="openshift-ingress-canary/ingress-canary-5ltrc"
Sep 15 07:18:50.043772 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:50.043667    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46hq6\" (UniqueName: \"kubernetes.io/projected/64a7e365-d705-4174-992f-bd271f4b418e-kube-api-access-46hq6\") pod \"dns-default-qvk9j\" (UID: \"64a7e365-d705-4174-992f-bd271f4b418e\") " pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:18:50.043772 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:50.043706    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fksm\" (UniqueName: \"kubernetes.io/projected/1934978a-de53-4efa-8cb8-2f2bc312ceaf-kube-api-access-6fksm\") pod \"ingress-canary-5ltrc\" (UID: \"1934978a-de53-4efa-8cb8-2f2bc312ceaf\") " pod="openshift-ingress-canary/ingress-canary-5ltrc"
Sep 15 07:18:50.043866 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:50.043768    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/64a7e365-d705-4174-992f-bd271f4b418e-config-volume\") pod \"dns-default-qvk9j\" (UID: \"64a7e365-d705-4174-992f-bd271f4b418e\") " pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:18:50.043866 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:50.043795    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls\") pod \"dns-default-qvk9j\" (UID: \"64a7e365-d705-4174-992f-bd271f4b418e\") " pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:18:50.043866 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:50.043832    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64a7e365-d705-4174-992f-bd271f4b418e-tmp-dir\") pod \"dns-default-qvk9j\" (UID: \"64a7e365-d705-4174-992f-bd271f4b418e\") " pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:18:50.044003 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:50.043905    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 07:18:50.044003 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:50.043972    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls podName:64a7e365-d705-4174-992f-bd271f4b418e nodeName:}" failed. No retries permitted until 2026-09-15 07:18:50.543952696 +0000 UTC m=+35.090522787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls") pod "dns-default-qvk9j" (UID: "64a7e365-d705-4174-992f-bd271f4b418e") : secret "dns-default-metrics-tls" not found
Sep 15 07:18:50.044239 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:50.044220    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/64a7e365-d705-4174-992f-bd271f4b418e-config-volume\") pod \"dns-default-qvk9j\" (UID: \"64a7e365-d705-4174-992f-bd271f4b418e\") " pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:18:50.046170 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:50.046152    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1934978a-de53-4efa-8cb8-2f2bc312ceaf-cert\") pod \"ingress-canary-5ltrc\" (UID: \"1934978a-de53-4efa-8cb8-2f2bc312ceaf\") " pod="openshift-ingress-canary/ingress-canary-5ltrc"
Sep 15 07:18:50.055515 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:50.055491    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46hq6\" (UniqueName: \"kubernetes.io/projected/64a7e365-d705-4174-992f-bd271f4b418e-kube-api-access-46hq6\") pod \"dns-default-qvk9j\" (UID: \"64a7e365-d705-4174-992f-bd271f4b418e\") " pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:18:50.055613 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:50.055525    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fksm\" (UniqueName: \"kubernetes.io/projected/1934978a-de53-4efa-8cb8-2f2bc312ceaf-kube-api-access-6fksm\") pod \"ingress-canary-5ltrc\" (UID: \"1934978a-de53-4efa-8cb8-2f2bc312ceaf\") " pod="openshift-ingress-canary/ingress-canary-5ltrc"
Sep 15 07:18:50.091892 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:50.091857    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5ltrc"
Sep 15 07:18:50.385540 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:50.385465    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5ltrc"]
Sep 15 07:18:50.389038 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:18:50.389014    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1934978a_de53_4efa_8cb8_2f2bc312ceaf.slice/crio-903090a6b7cb6f7c4e40dafc112af81c672025e3a23a1df4d601cbeffe3ae2ba WatchSource:0}: Error finding container 903090a6b7cb6f7c4e40dafc112af81c672025e3a23a1df4d601cbeffe3ae2ba: Status 404 returned error can't find the container with id 903090a6b7cb6f7c4e40dafc112af81c672025e3a23a1df4d601cbeffe3ae2ba
Sep 15 07:18:50.447191 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:50.447147    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-wdl2x\" (UID: \"6a8acf07-ffd3-4101-a1d5-16770855954c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x"
Sep 15 07:18:50.447191 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:50.447197    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:50.447412 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:50.447309    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 07:18:50.447412 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:50.447355    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 07:18:50.447412 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:50.447367    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bd85df4cd-n9nj4: secret "image-registry-tls" not found
Sep 15 07:18:50.447412 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:50.447410    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert podName:6a8acf07-ffd3-4101-a1d5-16770855954c nodeName:}" failed. No retries permitted until 2026-09-15 07:18:51.447387396 +0000 UTC m=+35.993957488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-wdl2x" (UID: "6a8acf07-ffd3-4101-a1d5-16770855954c") : secret "networking-console-plugin-cert" not found
Sep 15 07:18:50.447623 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:50.447429    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls podName:b8697c4f-de34-4c3f-ae67-fbbb429010c0 nodeName:}" failed. No retries permitted until 2026-09-15 07:18:51.447419921 +0000 UTC m=+35.993990003 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls") pod "image-registry-5bd85df4cd-n9nj4" (UID: "b8697c4f-de34-4c3f-ae67-fbbb429010c0") : secret "image-registry-tls" not found
Sep 15 07:18:50.548267 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:50.548231    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls\") pod \"dns-default-qvk9j\" (UID: \"64a7e365-d705-4174-992f-bd271f4b418e\") " pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:18:50.548423 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:50.548381    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 07:18:50.548459 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:50.548446    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls podName:64a7e365-d705-4174-992f-bd271f4b418e nodeName:}" failed. No retries permitted until 2026-09-15 07:18:51.548430953 +0000 UTC m=+36.095001048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls") pod "dns-default-qvk9j" (UID: "64a7e365-d705-4174-992f-bd271f4b418e") : secret "dns-default-metrics-tls" not found
Sep 15 07:18:50.903221 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:50.903184    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:18:50.906096 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:50.906074    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d7gwj\""
Sep 15 07:18:50.906239 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:50.906075    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 07:18:50.906239 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:50.906078    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 07:18:51.146556 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:51.146512    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5ltrc" event={"ID":"1934978a-de53-4efa-8cb8-2f2bc312ceaf","Type":"ContainerStarted","Data":"903090a6b7cb6f7c4e40dafc112af81c672025e3a23a1df4d601cbeffe3ae2ba"}
Sep 15 07:18:51.457235 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:51.457201    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-wdl2x\" (UID: \"6a8acf07-ffd3-4101-a1d5-16770855954c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x"
Sep 15 07:18:51.457437 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:51.457269    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:51.457437 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:51.457374    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 07:18:51.457437 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:51.457396    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 07:18:51.457437 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:51.457411    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bd85df4cd-n9nj4: secret "image-registry-tls" not found
Sep 15 07:18:51.457594 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:51.457456    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert podName:6a8acf07-ffd3-4101-a1d5-16770855954c nodeName:}" failed. No retries permitted until 2026-09-15 07:18:53.457435322 +0000 UTC m=+38.004005414 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-wdl2x" (UID: "6a8acf07-ffd3-4101-a1d5-16770855954c") : secret "networking-console-plugin-cert" not found
Sep 15 07:18:51.457594 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:51.457476    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls podName:b8697c4f-de34-4c3f-ae67-fbbb429010c0 nodeName:}" failed. No retries permitted until 2026-09-15 07:18:53.457466172 +0000 UTC m=+38.004036254 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls") pod "image-registry-5bd85df4cd-n9nj4" (UID: "b8697c4f-de34-4c3f-ae67-fbbb429010c0") : secret "image-registry-tls" not found
Sep 15 07:18:51.558573 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:51.558536    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls\") pod \"dns-default-qvk9j\" (UID: \"64a7e365-d705-4174-992f-bd271f4b418e\") " pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:18:51.558753 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:51.558704    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 07:18:51.558811 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:51.558781    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls podName:64a7e365-d705-4174-992f-bd271f4b418e nodeName:}" failed. No retries permitted until 2026-09-15 07:18:53.558761888 +0000 UTC m=+38.105331984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls") pod "dns-default-qvk9j" (UID: "64a7e365-d705-4174-992f-bd271f4b418e") : secret "dns-default-metrics-tls" not found
Sep 15 07:18:53.154287 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:53.154233    2571 generic.go:356] "Generic (PLEG): container finished" podID="542e179d-f430-4b8c-83f9-a7b45998ae91" containerID="3addaf43ba69c675ae60d094a90ef0350fc3011b298614bfb7ba5b0b0f57cb16" exitCode=0
Sep 15 07:18:53.154798 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:53.154287    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r247m" event={"ID":"542e179d-f430-4b8c-83f9-a7b45998ae91","Type":"ContainerDied","Data":"3addaf43ba69c675ae60d094a90ef0350fc3011b298614bfb7ba5b0b0f57cb16"}
Sep 15 07:18:53.476006 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:53.475783    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-wdl2x\" (UID: \"6a8acf07-ffd3-4101-a1d5-16770855954c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x"
Sep 15 07:18:53.476172 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:53.476030    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:53.476172 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:53.475936    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 07:18:53.476172 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:53.476154    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert podName:6a8acf07-ffd3-4101-a1d5-16770855954c nodeName:}" failed. No retries permitted until 2026-09-15 07:18:57.476132447 +0000 UTC m=+42.022702561 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-wdl2x" (UID: "6a8acf07-ffd3-4101-a1d5-16770855954c") : secret "networking-console-plugin-cert" not found
Sep 15 07:18:53.476362 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:53.476198    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 07:18:53.476362 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:53.476215    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bd85df4cd-n9nj4: secret "image-registry-tls" not found
Sep 15 07:18:53.476362 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:53.476286    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls podName:b8697c4f-de34-4c3f-ae67-fbbb429010c0 nodeName:}" failed. No retries permitted until 2026-09-15 07:18:57.476268623 +0000 UTC m=+42.022838726 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls") pod "image-registry-5bd85df4cd-n9nj4" (UID: "b8697c4f-de34-4c3f-ae67-fbbb429010c0") : secret "image-registry-tls" not found
Sep 15 07:18:53.577492 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:53.577446    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls\") pod \"dns-default-qvk9j\" (UID: \"64a7e365-d705-4174-992f-bd271f4b418e\") " pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:18:53.577639 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:53.577589    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 07:18:53.577678 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:53.577653    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls podName:64a7e365-d705-4174-992f-bd271f4b418e nodeName:}" failed. No retries permitted until 2026-09-15 07:18:57.57763828 +0000 UTC m=+42.124208358 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls") pod "dns-default-qvk9j" (UID: "64a7e365-d705-4174-992f-bd271f4b418e") : secret "dns-default-metrics-tls" not found
Sep 15 07:18:54.161178 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:54.161089    2571 generic.go:356] "Generic (PLEG): container finished" podID="542e179d-f430-4b8c-83f9-a7b45998ae91" containerID="faa2b93b3e5acfd1ea869b4f004311e3d12a285687ce1e781f988d2d30ef757f" exitCode=0
Sep 15 07:18:54.161178 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:54.161154    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r247m" event={"ID":"542e179d-f430-4b8c-83f9-a7b45998ae91","Type":"ContainerDied","Data":"faa2b93b3e5acfd1ea869b4f004311e3d12a285687ce1e781f988d2d30ef757f"}
Sep 15 07:18:55.167341 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:55.167235    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r247m" event={"ID":"542e179d-f430-4b8c-83f9-a7b45998ae91","Type":"ContainerStarted","Data":"74327e973d938b45d5304d818a669fcfadacb5da19cd9ab4af5b04772594a35b"}
Sep 15 07:18:55.168744 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:55.168723    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5ltrc" event={"ID":"1934978a-de53-4efa-8cb8-2f2bc312ceaf","Type":"ContainerStarted","Data":"07062a44157ad7a64b8578ecdc0e3df149233fd1bd88aad3ec80a1dfd18af7d3"}
Sep 15 07:18:55.188928 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:55.188880    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-r247m" podStartSLOduration=7.511869133 podStartE2EDuration="39.188862577s" totalImagesPullingTime="31.676993444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:18:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:18:55.187537228 +0000 UTC m=+39.734107338" watchObservedRunningTime="2026-09-15 07:18:55.188862577 +0000 UTC m=+39.735432676"
Sep 15 07:18:55.206795 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:55.206732    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5ltrc" podStartSLOduration=1.8297274369999998 podStartE2EDuration="6.20671218s" totalImagesPullingTime="4.376984743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:18:55.203686664 +0000 UTC m=+39.750256763" watchObservedRunningTime="2026-09-15 07:18:55.20671218 +0000 UTC m=+39.753282284"
Sep 15 07:18:57.508555 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:57.508512    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-wdl2x\" (UID: \"6a8acf07-ffd3-4101-a1d5-16770855954c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x"
Sep 15 07:18:57.508555 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:57.508557    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:18:57.508963 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:57.508662    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 07:18:57.508963 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:57.508662    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 07:18:57.508963 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:57.508731    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert podName:6a8acf07-ffd3-4101-a1d5-16770855954c nodeName:}" failed. No retries permitted until 2026-09-15 07:19:05.508715363 +0000 UTC m=+50.055285440 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-wdl2x" (UID: "6a8acf07-ffd3-4101-a1d5-16770855954c") : secret "networking-console-plugin-cert" not found
Sep 15 07:18:57.508963 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:57.508672    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bd85df4cd-n9nj4: secret "image-registry-tls" not found
Sep 15 07:18:57.508963 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:57.508796    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls podName:b8697c4f-de34-4c3f-ae67-fbbb429010c0 nodeName:}" failed. No retries permitted until 2026-09-15 07:19:05.50878506 +0000 UTC m=+50.055355136 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls") pod "image-registry-5bd85df4cd-n9nj4" (UID: "b8697c4f-de34-4c3f-ae67-fbbb429010c0") : secret "image-registry-tls" not found
Sep 15 07:18:57.609777 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:18:57.609740    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls\") pod \"dns-default-qvk9j\" (UID: \"64a7e365-d705-4174-992f-bd271f4b418e\") " pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:18:57.609922 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:57.609847    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 07:18:57.609922 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:18:57.609898    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls podName:64a7e365-d705-4174-992f-bd271f4b418e nodeName:}" failed. No retries permitted until 2026-09-15 07:19:05.609885231 +0000 UTC m=+50.156455309 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls") pod "dns-default-qvk9j" (UID: "64a7e365-d705-4174-992f-bd271f4b418e") : secret "dns-default-metrics-tls" not found
Sep 15 07:19:05.568475 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:05.568432    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-wdl2x\" (UID: \"6a8acf07-ffd3-4101-a1d5-16770855954c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x"
Sep 15 07:19:05.568475 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:05.568477    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:19:05.568917 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:05.568592    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 07:19:05.568917 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:05.568661    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert podName:6a8acf07-ffd3-4101-a1d5-16770855954c nodeName:}" failed. No retries permitted until 2026-09-15 07:19:21.568645592 +0000 UTC m=+66.115215683 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-wdl2x" (UID: "6a8acf07-ffd3-4101-a1d5-16770855954c") : secret "networking-console-plugin-cert" not found
Sep 15 07:19:05.568917 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:05.568596    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 07:19:05.568917 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:05.568684    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bd85df4cd-n9nj4: secret "image-registry-tls" not found
Sep 15 07:19:05.568917 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:05.568720    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls podName:b8697c4f-de34-4c3f-ae67-fbbb429010c0 nodeName:}" failed. No retries permitted until 2026-09-15 07:19:21.568710364 +0000 UTC m=+66.115280458 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls") pod "image-registry-5bd85df4cd-n9nj4" (UID: "b8697c4f-de34-4c3f-ae67-fbbb429010c0") : secret "image-registry-tls" not found
Sep 15 07:19:05.668803 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:05.668764    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls\") pod \"dns-default-qvk9j\" (UID: \"64a7e365-d705-4174-992f-bd271f4b418e\") " pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:19:05.668960 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:05.668911    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 07:19:05.669005 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:05.668972    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls podName:64a7e365-d705-4174-992f-bd271f4b418e nodeName:}" failed. No retries permitted until 2026-09-15 07:19:21.668955256 +0000 UTC m=+66.215525336 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls") pod "dns-default-qvk9j" (UID: "64a7e365-d705-4174-992f-bd271f4b418e") : secret "dns-default-metrics-tls" not found
Sep 15 07:19:16.140344 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:16.140315    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtrll"
Sep 15 07:19:21.583460 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:21.583403    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret\") pod \"global-pull-secret-syncer-kzskh\" (UID: \"08733b0b-6ce9-4bc5-87c5-4565bf0833b5\") " pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:19:21.583925 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:21.583506    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-wdl2x\" (UID: \"6a8acf07-ffd3-4101-a1d5-16770855954c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x"
Sep 15 07:19:21.583925 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:21.583537    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:19:21.583925 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:21.583640    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 07:19:21.583925 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:21.583644    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 07:19:21.583925 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:21.583721    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert podName:6a8acf07-ffd3-4101-a1d5-16770855954c nodeName:}" failed. No retries permitted until 2026-09-15 07:19:53.583704865 +0000 UTC m=+98.130274947 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-wdl2x" (UID: "6a8acf07-ffd3-4101-a1d5-16770855954c") : secret "networking-console-plugin-cert" not found
Sep 15 07:19:21.583925 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:21.583655    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bd85df4cd-n9nj4: secret "image-registry-tls" not found
Sep 15 07:19:21.583925 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:21.583807    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls podName:b8697c4f-de34-4c3f-ae67-fbbb429010c0 nodeName:}" failed. No retries permitted until 2026-09-15 07:19:53.583792069 +0000 UTC m=+98.130362148 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls") pod "image-registry-5bd85df4cd-n9nj4" (UID: "b8697c4f-de34-4c3f-ae67-fbbb429010c0") : secret "image-registry-tls" not found
Sep 15 07:19:21.586408 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:21.586390    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 07:19:21.597387 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:21.597368    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08733b0b-6ce9-4bc5-87c5-4565bf0833b5-original-pull-secret\") pod \"global-pull-secret-syncer-kzskh\" (UID: \"08733b0b-6ce9-4bc5-87c5-4565bf0833b5\") " pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:19:21.684751 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:21.684715    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls\") pod \"dns-default-qvk9j\" (UID: \"64a7e365-d705-4174-992f-bd271f4b418e\") " pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:19:21.684915 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:21.684772    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5kzw\" (UniqueName: \"kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw\") pod \"network-check-target-rfcdm\" (UID: \"7cc100f6-dbfe-412b-9ce2-845dea18988a\") " pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:19:21.684915 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:21.684819    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs\") pod \"network-metrics-daemon-jhh6r\" (UID: \"bf0b2425-be3b-4095-b8ab-a44f426176ac\") " pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:19:21.684915 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:21.684859    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 07:19:21.685063 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:21.684922    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls podName:64a7e365-d705-4174-992f-bd271f4b418e nodeName:}" failed. No retries permitted until 2026-09-15 07:19:53.684905881 +0000 UTC m=+98.231475976 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls") pod "dns-default-qvk9j" (UID: "64a7e365-d705-4174-992f-bd271f4b418e") : secret "dns-default-metrics-tls" not found
Sep 15 07:19:21.687340 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:21.687320    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 07:19:21.687391 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:21.687335    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 07:19:21.695416 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:21.695396    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 07:19:21.695469 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:21.695454    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs podName:bf0b2425-be3b-4095-b8ab-a44f426176ac nodeName:}" failed. No retries permitted until 2026-09-15 07:20:25.695437136 +0000 UTC m=+130.242007227 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs") pod "network-metrics-daemon-jhh6r" (UID: "bf0b2425-be3b-4095-b8ab-a44f426176ac") : secret "metrics-daemon-secret" not found
Sep 15 07:19:21.697206 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:21.697192    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 07:19:21.707924 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:21.707905    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5kzw\" (UniqueName: \"kubernetes.io/projected/7cc100f6-dbfe-412b-9ce2-845dea18988a-kube-api-access-s5kzw\") pod \"network-check-target-rfcdm\" (UID: \"7cc100f6-dbfe-412b-9ce2-845dea18988a\") " pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:19:21.721011 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:21.720991    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzskh"
Sep 15 07:19:21.816398 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:21.816365    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d7gwj\""
Sep 15 07:19:21.824743 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:21.824725    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:19:21.984342 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:21.984299    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kzskh"]
Sep 15 07:19:21.988720 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:19:21.988690    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08733b0b_6ce9_4bc5_87c5_4565bf0833b5.slice/crio-31189e1ac93b8dc692b37e5d7ff45c3b6fa84b6d99a15a7925d6c1ecb4d1e28f WatchSource:0}: Error finding container 31189e1ac93b8dc692b37e5d7ff45c3b6fa84b6d99a15a7925d6c1ecb4d1e28f: Status 404 returned error can't find the container with id 31189e1ac93b8dc692b37e5d7ff45c3b6fa84b6d99a15a7925d6c1ecb4d1e28f
Sep 15 07:19:22.087839 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:22.087803    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rfcdm"]
Sep 15 07:19:22.090647 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:19:22.090624    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc100f6_dbfe_412b_9ce2_845dea18988a.slice/crio-efac2f0a0118be0cc2acc79258b1c1dd37f6d84060460673d67981816a266b28 WatchSource:0}: Error finding container efac2f0a0118be0cc2acc79258b1c1dd37f6d84060460673d67981816a266b28: Status 404 returned error can't find the container with id efac2f0a0118be0cc2acc79258b1c1dd37f6d84060460673d67981816a266b28
Sep 15 07:19:22.234359 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:22.234326    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rfcdm" event={"ID":"7cc100f6-dbfe-412b-9ce2-845dea18988a","Type":"ContainerStarted","Data":"efac2f0a0118be0cc2acc79258b1c1dd37f6d84060460673d67981816a266b28"}
Sep 15 07:19:22.235326 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:22.235306    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kzskh" event={"ID":"08733b0b-6ce9-4bc5-87c5-4565bf0833b5","Type":"ContainerStarted","Data":"31189e1ac93b8dc692b37e5d7ff45c3b6fa84b6d99a15a7925d6c1ecb4d1e28f"}
Sep 15 07:19:27.250659 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:27.250618    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rfcdm" event={"ID":"7cc100f6-dbfe-412b-9ce2-845dea18988a","Type":"ContainerStarted","Data":"c1d15240fea6e2adee821ba40b49f7ecd78ce04ccb665073c6d9cb817ffb39ec"}
Sep 15 07:19:27.251095 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:27.250708    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:19:27.252077 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:27.252051    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kzskh" event={"ID":"08733b0b-6ce9-4bc5-87c5-4565bf0833b5","Type":"ContainerStarted","Data":"f38a194075832650c34ca4f6b7cca58725f7b73539592751174a9b9c15227256"}
Sep 15 07:19:27.277599 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:27.277555    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kzskh" podStartSLOduration=66.53511859299999 podStartE2EDuration="1m11.277540885s" totalImagesPullingTime="4.742422292s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:19:27.277110321 +0000 UTC m=+71.823680441" watchObservedRunningTime="2026-09-15 07:19:27.277540885 +0000 UTC m=+71.824110983"
Sep 15 07:19:27.277729 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:27.277696    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rfcdm" podStartSLOduration=66.641215707 podStartE2EDuration="1m11.27769129s" totalImagesPullingTime="4.636475583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:19:27.264910391 +0000 UTC m=+71.811480515" watchObservedRunningTime="2026-09-15 07:19:27.27769129 +0000 UTC m=+71.824261406"
Sep 15 07:19:53.616783 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:53.616736    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-wdl2x\" (UID: \"6a8acf07-ffd3-4101-a1d5-16770855954c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x"
Sep 15 07:19:53.616783 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:53.616782    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:19:53.617312 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:53.616875    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 07:19:53.617312 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:53.616879    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 07:19:53.617312 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:53.616945    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert podName:6a8acf07-ffd3-4101-a1d5-16770855954c nodeName:}" failed. No retries permitted until 2026-09-15 07:20:57.616929889 +0000 UTC m=+162.163499989 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-wdl2x" (UID: "6a8acf07-ffd3-4101-a1d5-16770855954c") : secret "networking-console-plugin-cert" not found
Sep 15 07:19:53.617312 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:53.616887    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bd85df4cd-n9nj4: secret "image-registry-tls" not found
Sep 15 07:19:53.617312 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:53.617014    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls podName:b8697c4f-de34-4c3f-ae67-fbbb429010c0 nodeName:}" failed. No retries permitted until 2026-09-15 07:20:57.617000574 +0000 UTC m=+162.163570651 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls") pod "image-registry-5bd85df4cd-n9nj4" (UID: "b8697c4f-de34-4c3f-ae67-fbbb429010c0") : secret "image-registry-tls" not found
Sep 15 07:19:53.717069 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:53.717037    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls\") pod \"dns-default-qvk9j\" (UID: \"64a7e365-d705-4174-992f-bd271f4b418e\") " pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:19:53.717213 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:53.717186    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 07:19:53.717267 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:19:53.717248    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls podName:64a7e365-d705-4174-992f-bd271f4b418e nodeName:}" failed. No retries permitted until 2026-09-15 07:20:57.717233525 +0000 UTC m=+162.263803601 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls") pod "dns-default-qvk9j" (UID: "64a7e365-d705-4174-992f-bd271f4b418e") : secret "dns-default-metrics-tls" not found
Sep 15 07:19:58.256376 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:19:58.256345    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rfcdm"
Sep 15 07:20:25.741101 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:20:25.741050    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs\") pod \"network-metrics-daemon-jhh6r\" (UID: \"bf0b2425-be3b-4095-b8ab-a44f426176ac\") " pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:20:25.741612 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:20:25.741191    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 07:20:25.741612 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:20:25.741283    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs podName:bf0b2425-be3b-4095-b8ab-a44f426176ac nodeName:}" failed. No retries permitted until 2026-09-15 07:22:27.741266576 +0000 UTC m=+252.287836667 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs") pod "network-metrics-daemon-jhh6r" (UID: "bf0b2425-be3b-4095-b8ab-a44f426176ac") : secret "metrics-daemon-secret" not found
Sep 15 07:20:52.759602 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:20:52.759548    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x" podUID="6a8acf07-ffd3-4101-a1d5-16770855954c"
Sep 15 07:20:52.764724 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:20:52.764686    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4" podUID="b8697c4f-de34-4c3f-ae67-fbbb429010c0"
Sep 15 07:20:52.784933 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:20:52.784897    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-qvk9j" podUID="64a7e365-d705-4174-992f-bd271f4b418e"
Sep 15 07:20:52.914700 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:20:52.914642    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-jhh6r" podUID="bf0b2425-be3b-4095-b8ab-a44f426176ac"
Sep 15 07:20:53.457126 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:20:53.457090    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:20:53.457337 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:20:53.457090    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x"
Sep 15 07:20:53.457337 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:20:53.457090    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:20:57.662556 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:20:57.662502    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-wdl2x\" (UID: \"6a8acf07-ffd3-4101-a1d5-16770855954c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x"
Sep 15 07:20:57.662556 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:20:57.662559    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:20:57.663106 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:20:57.662652    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 07:20:57.663106 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:20:57.662725    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert podName:6a8acf07-ffd3-4101-a1d5-16770855954c nodeName:}" failed. No retries permitted until 2026-09-15 07:22:59.662709428 +0000 UTC m=+284.209279521 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-wdl2x" (UID: "6a8acf07-ffd3-4101-a1d5-16770855954c") : secret "networking-console-plugin-cert" not found
Sep 15 07:20:57.663106 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:20:57.662653    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 07:20:57.663106 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:20:57.662763    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bd85df4cd-n9nj4: secret "image-registry-tls" not found
Sep 15 07:20:57.663106 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:20:57.662806    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls podName:b8697c4f-de34-4c3f-ae67-fbbb429010c0 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:59.662793722 +0000 UTC m=+284.209363799 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls") pod "image-registry-5bd85df4cd-n9nj4" (UID: "b8697c4f-de34-4c3f-ae67-fbbb429010c0") : secret "image-registry-tls" not found
Sep 15 07:20:57.763830 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:20:57.763791    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls\") pod \"dns-default-qvk9j\" (UID: \"64a7e365-d705-4174-992f-bd271f4b418e\") " pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:20:57.763966 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:20:57.763935    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 07:20:57.764003 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:20:57.763994    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls podName:64a7e365-d705-4174-992f-bd271f4b418e nodeName:}" failed. No retries permitted until 2026-09-15 07:22:59.76397734 +0000 UTC m=+284.310547425 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls") pod "dns-default-qvk9j" (UID: "64a7e365-d705-4174-992f-bd271f4b418e") : secret "dns-default-metrics-tls" not found
Sep 15 07:21:05.905035 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:21:05.904997    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:22:26.978522 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:26.978490    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7kvwg"]
Sep 15 07:22:26.980364 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:26.980348    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7kvwg"
Sep 15 07:22:26.982467 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:26.982441    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-j7dlf"]
Sep 15 07:22:26.982793 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:26.982773    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-x7cpw\""
Sep 15 07:22:26.982877 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:26.982773    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 15 07:22:26.982877 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:26.982858    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 15 07:22:26.984936 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:26.984920    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp"]
Sep 15 07:22:26.985052 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:26.985035    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-j7dlf"
Sep 15 07:22:26.986960 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:26.986940    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp"
Sep 15 07:22:26.987842 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:26.987435    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 15 07:22:26.987842 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:26.987464    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 15 07:22:26.988154 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:26.988135    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 15 07:22:26.988235 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:26.988183    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-k24w9\""
Sep 15 07:22:26.988235 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:26.988141    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 15 07:22:26.991788 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:26.991752    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 07:22:26.992453 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:26.992431    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7kvwg"]
Sep 15 07:22:26.993066 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:26.993046    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 07:22:26.993066 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:26.993057    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bf5jz\""
Sep 15 07:22:26.993268 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:26.993097    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 15 07:22:26.993334 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:26.993325    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 15 07:22:26.998447 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:26.998429    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-j7dlf"]
Sep 15 07:22:27.001905 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.001880    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp"]
Sep 15 07:22:27.073725 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.073693    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-j5lqb"]
Sep 15 07:22:27.075931 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.075914    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-j5lqb"
Sep 15 07:22:27.078052 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.078024    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7lzrk\""
Sep 15 07:22:27.082114 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.082087    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-kc5ms"]
Sep 15 07:22:27.083779 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.083761    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c85gj"]
Sep 15 07:22:27.083904 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.083889    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kc5ms"
Sep 15 07:22:27.085747 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.085729    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c85gj"
Sep 15 07:22:27.087136 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.087121    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 15 07:22:27.087215 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.087158    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 15 07:22:27.087649 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.087631    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 15 07:22:27.088290 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.088274    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-knb4d\""
Sep 15 07:22:27.090982 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.090959    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-j5lqb"]
Sep 15 07:22:27.092236 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.092206    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 15 07:22:27.092348 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.092264    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-dvhrj\""
Sep 15 07:22:27.092348 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.092290    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 15 07:22:27.092348 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.092299    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 15 07:22:27.092563 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.092547    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 15 07:22:27.104729 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.104710    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-kc5ms"]
Sep 15 07:22:27.105434 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.105417    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c85gj"]
Sep 15 07:22:27.174199 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.174166    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea4c7c35-ce08-453e-b629-f52c0ab26c66-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-j7dlf\" (UID: \"ea4c7c35-ce08-453e-b629-f52c0ab26c66\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-j7dlf"
Sep 15 07:22:27.174378 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.174211    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxvsg\" (UniqueName: \"kubernetes.io/projected/ea4c7c35-ce08-453e-b629-f52c0ab26c66-kube-api-access-hxvsg\") pod \"kube-storage-version-migrator-operator-6c8f84f897-j7dlf\" (UID: \"ea4c7c35-ce08-453e-b629-f52c0ab26c66\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-j7dlf"
Sep 15 07:22:27.174378 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.174305    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lk8h\" (UniqueName: \"kubernetes.io/projected/1891cbd6-62bb-4387-8822-552d9047b120-kube-api-access-2lk8h\") pod \"volume-data-source-validator-86695b7cc7-7kvwg\" (UID: \"1891cbd6-62bb-4387-8822-552d9047b120\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7kvwg"
Sep 15 07:22:27.174378 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.174336    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ea4c7c35-ce08-453e-b629-f52c0ab26c66-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-j7dlf\" (UID: \"ea4c7c35-ce08-453e-b629-f52c0ab26c66\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-j7dlf"
Sep 15 07:22:27.174378 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.174363    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f86f2096-0758-472d-a436-90432eb7368c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-v6vxp\" (UID: \"f86f2096-0758-472d-a436-90432eb7368c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp"
Sep 15 07:22:27.174528 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.174379    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdfhk\" (UniqueName: \"kubernetes.io/projected/db358ba6-5910-4fb5-850e-11aa959e024c-kube-api-access-cdfhk\") pod \"network-check-source-6f8b768455-j5lqb\" (UID: \"db358ba6-5910-4fb5-850e-11aa959e024c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-j5lqb"
Sep 15 07:22:27.174528 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.174407    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f86f2096-0758-472d-a436-90432eb7368c-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-v6vxp\" (UID: \"f86f2096-0758-472d-a436-90432eb7368c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp"
Sep 15 07:22:27.174528 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.174421    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr5xw\" (UniqueName: \"kubernetes.io/projected/f86f2096-0758-472d-a436-90432eb7368c-kube-api-access-jr5xw\") pod \"cluster-monitoring-operator-6cd954b897-v6vxp\" (UID: \"f86f2096-0758-472d-a436-90432eb7368c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp"
Sep 15 07:22:27.275535 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.275452    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jmkn\" (UniqueName: \"kubernetes.io/projected/591b847d-1325-4654-aaa1-e2d142921b00-kube-api-access-2jmkn\") pod \"cluster-samples-operator-86499c89c4-c85gj\" (UID: \"591b847d-1325-4654-aaa1-e2d142921b00\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c85gj"
Sep 15 07:22:27.275535 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.275499    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw55v\" (UniqueName: \"kubernetes.io/projected/fb9280f4-141e-41f1-8e67-4d08778d723f-kube-api-access-jw55v\") pod \"service-ca-operator-cf7c776b7-kc5ms\" (UID: \"fb9280f4-141e-41f1-8e67-4d08778d723f\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kc5ms"
Sep 15 07:22:27.275692 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.275556    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxvsg\" (UniqueName: \"kubernetes.io/projected/ea4c7c35-ce08-453e-b629-f52c0ab26c66-kube-api-access-hxvsg\") pod \"kube-storage-version-migrator-operator-6c8f84f897-j7dlf\" (UID: \"ea4c7c35-ce08-453e-b629-f52c0ab26c66\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-j7dlf"
Sep 15 07:22:27.275692 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.275589    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f86f2096-0758-472d-a436-90432eb7368c-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-v6vxp\" (UID: \"f86f2096-0758-472d-a436-90432eb7368c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp"
Sep 15 07:22:27.275692 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.275607    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr5xw\" (UniqueName: \"kubernetes.io/projected/f86f2096-0758-472d-a436-90432eb7368c-kube-api-access-jr5xw\") pod \"cluster-monitoring-operator-6cd954b897-v6vxp\" (UID: \"f86f2096-0758-472d-a436-90432eb7368c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp"
Sep 15 07:22:27.275692 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.275624    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/591b847d-1325-4654-aaa1-e2d142921b00-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c85gj\" (UID: \"591b847d-1325-4654-aaa1-e2d142921b00\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c85gj"
Sep 15 07:22:27.275934 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.275733    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdfhk\" (UniqueName: \"kubernetes.io/projected/db358ba6-5910-4fb5-850e-11aa959e024c-kube-api-access-cdfhk\") pod \"network-check-source-6f8b768455-j5lqb\" (UID: \"db358ba6-5910-4fb5-850e-11aa959e024c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-j5lqb"
Sep 15 07:22:27.275934 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.275767    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb9280f4-141e-41f1-8e67-4d08778d723f-config\") pod \"service-ca-operator-cf7c776b7-kc5ms\" (UID: \"fb9280f4-141e-41f1-8e67-4d08778d723f\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kc5ms"
Sep 15 07:22:27.275934 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.275785    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea4c7c35-ce08-453e-b629-f52c0ab26c66-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-j7dlf\" (UID: \"ea4c7c35-ce08-453e-b629-f52c0ab26c66\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-j7dlf"
Sep 15 07:22:27.275934 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.275800    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb9280f4-141e-41f1-8e67-4d08778d723f-serving-cert\") pod \"service-ca-operator-cf7c776b7-kc5ms\" (UID: \"fb9280f4-141e-41f1-8e67-4d08778d723f\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kc5ms"
Sep 15 07:22:27.275934 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.275834    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lk8h\" (UniqueName: \"kubernetes.io/projected/1891cbd6-62bb-4387-8822-552d9047b120-kube-api-access-2lk8h\") pod \"volume-data-source-validator-86695b7cc7-7kvwg\" (UID: \"1891cbd6-62bb-4387-8822-552d9047b120\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7kvwg"
Sep 15 07:22:27.275934 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.275850    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ea4c7c35-ce08-453e-b629-f52c0ab26c66-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-j7dlf\" (UID: \"ea4c7c35-ce08-453e-b629-f52c0ab26c66\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-j7dlf"
Sep 15 07:22:27.275934 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.275879    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f86f2096-0758-472d-a436-90432eb7368c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-v6vxp\" (UID: \"f86f2096-0758-472d-a436-90432eb7368c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp"
Sep 15 07:22:27.276305 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:27.275991    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 07:22:27.276305 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:27.276049    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f86f2096-0758-472d-a436-90432eb7368c-cluster-monitoring-operator-tls podName:f86f2096-0758-472d-a436-90432eb7368c nodeName:}" failed. No retries permitted until 2026-09-15 07:22:27.776036302 +0000 UTC m=+252.322606379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f86f2096-0758-472d-a436-90432eb7368c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-v6vxp" (UID: "f86f2096-0758-472d-a436-90432eb7368c") : secret "cluster-monitoring-operator-tls" not found
Sep 15 07:22:27.276383 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.276316    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f86f2096-0758-472d-a436-90432eb7368c-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-v6vxp\" (UID: \"f86f2096-0758-472d-a436-90432eb7368c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp"
Sep 15 07:22:27.276448 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.276430    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ea4c7c35-ce08-453e-b629-f52c0ab26c66-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-j7dlf\" (UID: \"ea4c7c35-ce08-453e-b629-f52c0ab26c66\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-j7dlf"
Sep 15 07:22:27.278040 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.278019    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea4c7c35-ce08-453e-b629-f52c0ab26c66-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-j7dlf\" (UID: \"ea4c7c35-ce08-453e-b629-f52c0ab26c66\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-j7dlf"
Sep 15 07:22:27.285863 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.285836    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lk8h\" (UniqueName: \"kubernetes.io/projected/1891cbd6-62bb-4387-8822-552d9047b120-kube-api-access-2lk8h\") pod \"volume-data-source-validator-86695b7cc7-7kvwg\" (UID: \"1891cbd6-62bb-4387-8822-552d9047b120\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7kvwg"
Sep 15 07:22:27.288226 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.288198    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxvsg\" (UniqueName: \"kubernetes.io/projected/ea4c7c35-ce08-453e-b629-f52c0ab26c66-kube-api-access-hxvsg\") pod \"kube-storage-version-migrator-operator-6c8f84f897-j7dlf\" (UID: \"ea4c7c35-ce08-453e-b629-f52c0ab26c66\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-j7dlf"
Sep 15 07:22:27.288608 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.288591    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr5xw\" (UniqueName: \"kubernetes.io/projected/f86f2096-0758-472d-a436-90432eb7368c-kube-api-access-jr5xw\") pod \"cluster-monitoring-operator-6cd954b897-v6vxp\" (UID: \"f86f2096-0758-472d-a436-90432eb7368c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp"
Sep 15 07:22:27.288708 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.288686    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdfhk\" (UniqueName: \"kubernetes.io/projected/db358ba6-5910-4fb5-850e-11aa959e024c-kube-api-access-cdfhk\") pod \"network-check-source-6f8b768455-j5lqb\" (UID: \"db358ba6-5910-4fb5-850e-11aa959e024c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-j5lqb"
Sep 15 07:22:27.290370 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.290353    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7kvwg"
Sep 15 07:22:27.297839 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.297817    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-j7dlf"
Sep 15 07:22:27.376693 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.376664    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jmkn\" (UniqueName: \"kubernetes.io/projected/591b847d-1325-4654-aaa1-e2d142921b00-kube-api-access-2jmkn\") pod \"cluster-samples-operator-86499c89c4-c85gj\" (UID: \"591b847d-1325-4654-aaa1-e2d142921b00\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c85gj"
Sep 15 07:22:27.376811 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.376701    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw55v\" (UniqueName: \"kubernetes.io/projected/fb9280f4-141e-41f1-8e67-4d08778d723f-kube-api-access-jw55v\") pod \"service-ca-operator-cf7c776b7-kc5ms\" (UID: \"fb9280f4-141e-41f1-8e67-4d08778d723f\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kc5ms"
Sep 15 07:22:27.376855 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.376826    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/591b847d-1325-4654-aaa1-e2d142921b00-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c85gj\" (UID: \"591b847d-1325-4654-aaa1-e2d142921b00\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c85gj"
Sep 15 07:22:27.376917 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.376898    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb9280f4-141e-41f1-8e67-4d08778d723f-config\") pod \"service-ca-operator-cf7c776b7-kc5ms\" (UID: \"fb9280f4-141e-41f1-8e67-4d08778d723f\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kc5ms"
Sep 15 07:22:27.376973 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.376925    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb9280f4-141e-41f1-8e67-4d08778d723f-serving-cert\") pod \"service-ca-operator-cf7c776b7-kc5ms\" (UID: \"fb9280f4-141e-41f1-8e67-4d08778d723f\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kc5ms"
Sep 15 07:22:27.377021 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:27.376972    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 07:22:27.377069 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:27.377042    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/591b847d-1325-4654-aaa1-e2d142921b00-samples-operator-tls podName:591b847d-1325-4654-aaa1-e2d142921b00 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:27.877022167 +0000 UTC m=+252.423592252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/591b847d-1325-4654-aaa1-e2d142921b00-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-c85gj" (UID: "591b847d-1325-4654-aaa1-e2d142921b00") : secret "samples-operator-tls" not found
Sep 15 07:22:27.377488 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.377468    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb9280f4-141e-41f1-8e67-4d08778d723f-config\") pod \"service-ca-operator-cf7c776b7-kc5ms\" (UID: \"fb9280f4-141e-41f1-8e67-4d08778d723f\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kc5ms"
Sep 15 07:22:27.378992 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.378977    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb9280f4-141e-41f1-8e67-4d08778d723f-serving-cert\") pod \"service-ca-operator-cf7c776b7-kc5ms\" (UID: \"fb9280f4-141e-41f1-8e67-4d08778d723f\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kc5ms"
Sep 15 07:22:27.384188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.384169    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-j5lqb"
Sep 15 07:22:27.384946 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.384901    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw55v\" (UniqueName: \"kubernetes.io/projected/fb9280f4-141e-41f1-8e67-4d08778d723f-kube-api-access-jw55v\") pod \"service-ca-operator-cf7c776b7-kc5ms\" (UID: \"fb9280f4-141e-41f1-8e67-4d08778d723f\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kc5ms"
Sep 15 07:22:27.384946 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.384909    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jmkn\" (UniqueName: \"kubernetes.io/projected/591b847d-1325-4654-aaa1-e2d142921b00-kube-api-access-2jmkn\") pod \"cluster-samples-operator-86499c89c4-c85gj\" (UID: \"591b847d-1325-4654-aaa1-e2d142921b00\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c85gj"
Sep 15 07:22:27.393744 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.393725    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kc5ms"
Sep 15 07:22:27.559043 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.559018    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7kvwg"]
Sep 15 07:22:27.561910 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:22:27.561881    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1891cbd6_62bb_4387_8822_552d9047b120.slice/crio-8b0a67903fd091d572d290df8bf1a2ffa7737916ec7dd203b4e21f1d22c73905 WatchSource:0}: Error finding container 8b0a67903fd091d572d290df8bf1a2ffa7737916ec7dd203b4e21f1d22c73905: Status 404 returned error can't find the container with id 8b0a67903fd091d572d290df8bf1a2ffa7737916ec7dd203b4e21f1d22c73905
Sep 15 07:22:27.562769 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.562755    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-j7dlf"]
Sep 15 07:22:27.564951 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:22:27.564931    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4c7c35_ce08_453e_b629_f52c0ab26c66.slice/crio-a712fe25ea8d4c1b67975a1c67f6f12f45a8f884abbb62a548b160409142ea17 WatchSource:0}: Error finding container a712fe25ea8d4c1b67975a1c67f6f12f45a8f884abbb62a548b160409142ea17: Status 404 returned error can't find the container with id a712fe25ea8d4c1b67975a1c67f6f12f45a8f884abbb62a548b160409142ea17
Sep 15 07:22:27.654960 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.654935    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-j5lqb"]
Sep 15 07:22:27.657286 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:22:27.657242    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb358ba6_5910_4fb5_850e_11aa959e024c.slice/crio-7f25fa5dd07516c37ecfece8ed0d5881a756fdb35591feb829c85d3b2e8291da WatchSource:0}: Error finding container 7f25fa5dd07516c37ecfece8ed0d5881a756fdb35591feb829c85d3b2e8291da: Status 404 returned error can't find the container with id 7f25fa5dd07516c37ecfece8ed0d5881a756fdb35591feb829c85d3b2e8291da
Sep 15 07:22:27.676880 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.676849    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-j5lqb" event={"ID":"db358ba6-5910-4fb5-850e-11aa959e024c","Type":"ContainerStarted","Data":"7f25fa5dd07516c37ecfece8ed0d5881a756fdb35591feb829c85d3b2e8291da"}
Sep 15 07:22:27.677579 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.677557    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-kc5ms"]
Sep 15 07:22:27.678076 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.678058    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-j7dlf" event={"ID":"ea4c7c35-ce08-453e-b629-f52c0ab26c66","Type":"ContainerStarted","Data":"a712fe25ea8d4c1b67975a1c67f6f12f45a8f884abbb62a548b160409142ea17"}
Sep 15 07:22:27.679195 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.679177    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7kvwg" event={"ID":"1891cbd6-62bb-4387-8822-552d9047b120","Type":"ContainerStarted","Data":"8b0a67903fd091d572d290df8bf1a2ffa7737916ec7dd203b4e21f1d22c73905"}
Sep 15 07:22:27.686068 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:22:27.686047    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9280f4_141e_41f1_8e67_4d08778d723f.slice/crio-ed9e81418375e7e609f1e2bf314bb3951640b8f337c00d15d84f7a5c2cf4b89e WatchSource:0}: Error finding container ed9e81418375e7e609f1e2bf314bb3951640b8f337c00d15d84f7a5c2cf4b89e: Status 404 returned error can't find the container with id ed9e81418375e7e609f1e2bf314bb3951640b8f337c00d15d84f7a5c2cf4b89e
Sep 15 07:22:27.779436 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.779406    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f86f2096-0758-472d-a436-90432eb7368c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-v6vxp\" (UID: \"f86f2096-0758-472d-a436-90432eb7368c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp"
Sep 15 07:22:27.779570 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.779451    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs\") pod \"network-metrics-daemon-jhh6r\" (UID: \"bf0b2425-be3b-4095-b8ab-a44f426176ac\") " pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:22:27.779570 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:27.779551    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 07:22:27.779570 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:27.779554    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 07:22:27.779666 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:27.779598    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs podName:bf0b2425-be3b-4095-b8ab-a44f426176ac nodeName:}" failed. No retries permitted until 2026-09-15 07:24:29.779585632 +0000 UTC m=+374.326155721 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs") pod "network-metrics-daemon-jhh6r" (UID: "bf0b2425-be3b-4095-b8ab-a44f426176ac") : secret "metrics-daemon-secret" not found
Sep 15 07:22:27.779666 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:27.779610    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f86f2096-0758-472d-a436-90432eb7368c-cluster-monitoring-operator-tls podName:f86f2096-0758-472d-a436-90432eb7368c nodeName:}" failed. No retries permitted until 2026-09-15 07:22:28.779604536 +0000 UTC m=+253.326174613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f86f2096-0758-472d-a436-90432eb7368c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-v6vxp" (UID: "f86f2096-0758-472d-a436-90432eb7368c") : secret "cluster-monitoring-operator-tls" not found
Sep 15 07:22:27.879876 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:27.879783    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/591b847d-1325-4654-aaa1-e2d142921b00-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c85gj\" (UID: \"591b847d-1325-4654-aaa1-e2d142921b00\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c85gj"
Sep 15 07:22:27.880009 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:27.879936    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 07:22:27.880009 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:27.879999    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/591b847d-1325-4654-aaa1-e2d142921b00-samples-operator-tls podName:591b847d-1325-4654-aaa1-e2d142921b00 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:28.879981356 +0000 UTC m=+253.426551451 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/591b847d-1325-4654-aaa1-e2d142921b00-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-c85gj" (UID: "591b847d-1325-4654-aaa1-e2d142921b00") : secret "samples-operator-tls" not found
Sep 15 07:22:28.683178 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:28.683139    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-j5lqb" event={"ID":"db358ba6-5910-4fb5-850e-11aa959e024c","Type":"ContainerStarted","Data":"7feb0f81ffcaf61084795fbe8680c2b025b55be9bbded732dbf21a112432693f"}
Sep 15 07:22:28.684353 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:28.684326    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kc5ms" event={"ID":"fb9280f4-141e-41f1-8e67-4d08778d723f","Type":"ContainerStarted","Data":"ed9e81418375e7e609f1e2bf314bb3951640b8f337c00d15d84f7a5c2cf4b89e"}
Sep 15 07:22:28.701055 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:28.701009    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-j5lqb" podStartSLOduration=1.70099705 podStartE2EDuration="1.70099705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:22:28.700400112 +0000 UTC m=+253.246970233" watchObservedRunningTime="2026-09-15 07:22:28.70099705 +0000 UTC m=+253.247567149"
Sep 15 07:22:28.786191 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:28.786159    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f86f2096-0758-472d-a436-90432eb7368c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-v6vxp\" (UID: \"f86f2096-0758-472d-a436-90432eb7368c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp"
Sep 15 07:22:28.786349 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:28.786278    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 07:22:28.786349 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:28.786334    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f86f2096-0758-472d-a436-90432eb7368c-cluster-monitoring-operator-tls podName:f86f2096-0758-472d-a436-90432eb7368c nodeName:}" failed. No retries permitted until 2026-09-15 07:22:30.786321036 +0000 UTC m=+255.332891113 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f86f2096-0758-472d-a436-90432eb7368c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-v6vxp" (UID: "f86f2096-0758-472d-a436-90432eb7368c") : secret "cluster-monitoring-operator-tls" not found
Sep 15 07:22:28.887455 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:28.887415    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/591b847d-1325-4654-aaa1-e2d142921b00-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c85gj\" (UID: \"591b847d-1325-4654-aaa1-e2d142921b00\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c85gj"
Sep 15 07:22:28.887615 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:28.887543    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 07:22:28.887615 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:28.887594    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/591b847d-1325-4654-aaa1-e2d142921b00-samples-operator-tls podName:591b847d-1325-4654-aaa1-e2d142921b00 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:30.887581485 +0000 UTC m=+255.434151561 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/591b847d-1325-4654-aaa1-e2d142921b00-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-c85gj" (UID: "591b847d-1325-4654-aaa1-e2d142921b00") : secret "samples-operator-tls" not found
Sep 15 07:22:30.804843 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:30.804810    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f86f2096-0758-472d-a436-90432eb7368c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-v6vxp\" (UID: \"f86f2096-0758-472d-a436-90432eb7368c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp"
Sep 15 07:22:30.805222 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:30.804971    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 07:22:30.805222 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:30.805037    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f86f2096-0758-472d-a436-90432eb7368c-cluster-monitoring-operator-tls podName:f86f2096-0758-472d-a436-90432eb7368c nodeName:}" failed. No retries permitted until 2026-09-15 07:22:34.805016345 +0000 UTC m=+259.351586425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f86f2096-0758-472d-a436-90432eb7368c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-v6vxp" (UID: "f86f2096-0758-472d-a436-90432eb7368c") : secret "cluster-monitoring-operator-tls" not found
Sep 15 07:22:30.905590 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:30.905549    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/591b847d-1325-4654-aaa1-e2d142921b00-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c85gj\" (UID: \"591b847d-1325-4654-aaa1-e2d142921b00\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c85gj"
Sep 15 07:22:30.905754 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:30.905720    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 07:22:30.905796 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:30.905784    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/591b847d-1325-4654-aaa1-e2d142921b00-samples-operator-tls podName:591b847d-1325-4654-aaa1-e2d142921b00 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:34.905766894 +0000 UTC m=+259.452336976 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/591b847d-1325-4654-aaa1-e2d142921b00-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-c85gj" (UID: "591b847d-1325-4654-aaa1-e2d142921b00") : secret "samples-operator-tls" not found
Sep 15 07:22:31.410508 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.410471    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qk9cv"]
Sep 15 07:22:31.413481 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.413458    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:31.416025 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.416005    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 07:22:31.417053 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.417023    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 07:22:31.417053 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.417048    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mn7st\""
Sep 15 07:22:31.417222 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.417079    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 07:22:31.417222 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.417051    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 07:22:31.426628 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.426583    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qk9cv"]
Sep 15 07:22:31.510666 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.510625    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4adb8650-815d-427e-bf5e-f0fb7d41db60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:31.510867 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.510680    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4adb8650-815d-427e-bf5e-f0fb7d41db60-data-volume\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:31.510867 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.510715    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x7x7\" (UniqueName: \"kubernetes.io/projected/4adb8650-815d-427e-bf5e-f0fb7d41db60-kube-api-access-2x7x7\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:31.510867 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.510752    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4adb8650-815d-427e-bf5e-f0fb7d41db60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:31.510867 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.510825    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4adb8650-815d-427e-bf5e-f0fb7d41db60-crio-socket\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:31.611844 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.611819    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4adb8650-815d-427e-bf5e-f0fb7d41db60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:31.611945 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.611851    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4adb8650-815d-427e-bf5e-f0fb7d41db60-data-volume\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:31.611945 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.611870    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x7x7\" (UniqueName: \"kubernetes.io/projected/4adb8650-815d-427e-bf5e-f0fb7d41db60-kube-api-access-2x7x7\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:31.611945 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.611899    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4adb8650-815d-427e-bf5e-f0fb7d41db60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:31.612080 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:31.611947    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 07:22:31.612080 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:31.611990    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:31.612080 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:31.612018    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4adb8650-815d-427e-bf5e-f0fb7d41db60-kube-rbac-proxy-cm podName:4adb8650-815d-427e-bf5e-f0fb7d41db60 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:32.112000113 +0000 UTC m=+256.658570203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4adb8650-815d-427e-bf5e-f0fb7d41db60-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qk9cv" (UID: "4adb8650-815d-427e-bf5e-f0fb7d41db60") : configmap "kube-rbac-proxy" not found
Sep 15 07:22:31.612080 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:31.612039    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4adb8650-815d-427e-bf5e-f0fb7d41db60-insights-runtime-extractor-tls podName:4adb8650-815d-427e-bf5e-f0fb7d41db60 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:32.112029229 +0000 UTC m=+256.658599310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4adb8650-815d-427e-bf5e-f0fb7d41db60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qk9cv" (UID: "4adb8650-815d-427e-bf5e-f0fb7d41db60") : secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:31.612299 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.612179    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4adb8650-815d-427e-bf5e-f0fb7d41db60-data-volume\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:31.612299 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.612191    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4adb8650-815d-427e-bf5e-f0fb7d41db60-crio-socket\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:31.612299 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.612245    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4adb8650-815d-427e-bf5e-f0fb7d41db60-crio-socket\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:31.622662 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.622636    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x7x7\" (UniqueName: \"kubernetes.io/projected/4adb8650-815d-427e-bf5e-f0fb7d41db60-kube-api-access-2x7x7\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:31.696074 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.695980    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kc5ms" event={"ID":"fb9280f4-141e-41f1-8e67-4d08778d723f","Type":"ContainerStarted","Data":"3e36267b3230249c293c7d49708b2132e14825bf2baa16b19d530bb66203430f"}
Sep 15 07:22:31.697403 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.697377    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-j7dlf" event={"ID":"ea4c7c35-ce08-453e-b629-f52c0ab26c66","Type":"ContainerStarted","Data":"2169f093e51a106770f358c6a1eba3f66cd3daf0b88fd3922aa8931bb8ba20f7"}
Sep 15 07:22:31.698891 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.698830    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7kvwg" event={"ID":"1891cbd6-62bb-4387-8822-552d9047b120","Type":"ContainerStarted","Data":"3abf65e5b88c22f6bd447d8086dc99941e203c1491cfef6c28d1bb0b7b7a519d"}
Sep 15 07:22:31.712099 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.712054    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kc5ms" podStartSLOduration=0.850621061 podStartE2EDuration="4.712035808s" totalImagesPullingTime="3.861414747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:22:31.711099993 +0000 UTC m=+256.257670107" watchObservedRunningTime="2026-09-15 07:22:31.712035808 +0000 UTC m=+256.258605933"
Sep 15 07:22:31.728540 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.728408    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7kvwg" podStartSLOduration=2.535572086 podStartE2EDuration="5.728386476s" totalImagesPullingTime="3.19281439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:22:31.727489754 +0000 UTC m=+256.274059878" watchObservedRunningTime="2026-09-15 07:22:31.728386476 +0000 UTC m=+256.274956578"
Sep 15 07:22:31.748048 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.747997    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-j7dlf" podStartSLOduration=2.558884027 podStartE2EDuration="5.747978004s" totalImagesPullingTime="3.189093977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:22:31.746903151 +0000 UTC m=+256.293473272" watchObservedRunningTime="2026-09-15 07:22:31.747978004 +0000 UTC m=+256.294548099"
Sep 15 07:22:31.814827 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.814795    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-5rzzs"]
Sep 15 07:22:31.817126 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.817101    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-5rzzs"
Sep 15 07:22:31.819770 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.819747    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 15 07:22:31.819879 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.819833    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 15 07:22:31.819946 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.819912    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qhz76\""
Sep 15 07:22:31.826641 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.826619    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-5rzzs"]
Sep 15 07:22:31.915807 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:31.915775    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd7qw\" (UniqueName: \"kubernetes.io/projected/299e0037-3343-48d4-bb94-080df4472b59-kube-api-access-bd7qw\") pod \"migrator-8b4d697c5-5rzzs\" (UID: \"299e0037-3343-48d4-bb94-080df4472b59\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-5rzzs"
Sep 15 07:22:32.017072 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:32.017039    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd7qw\" (UniqueName: \"kubernetes.io/projected/299e0037-3343-48d4-bb94-080df4472b59-kube-api-access-bd7qw\") pod \"migrator-8b4d697c5-5rzzs\" (UID: \"299e0037-3343-48d4-bb94-080df4472b59\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-5rzzs"
Sep 15 07:22:32.024709 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:32.024684    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd7qw\" (UniqueName: \"kubernetes.io/projected/299e0037-3343-48d4-bb94-080df4472b59-kube-api-access-bd7qw\") pod \"migrator-8b4d697c5-5rzzs\" (UID: \"299e0037-3343-48d4-bb94-080df4472b59\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-5rzzs"
Sep 15 07:22:32.118058 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:32.118024    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4adb8650-815d-427e-bf5e-f0fb7d41db60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:32.118216 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:32.118078    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4adb8650-815d-427e-bf5e-f0fb7d41db60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:32.118216 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:32.118139    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 07:22:32.118216 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:32.118210    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4adb8650-815d-427e-bf5e-f0fb7d41db60-kube-rbac-proxy-cm podName:4adb8650-815d-427e-bf5e-f0fb7d41db60 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:33.118193332 +0000 UTC m=+257.664763414 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4adb8650-815d-427e-bf5e-f0fb7d41db60-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qk9cv" (UID: "4adb8650-815d-427e-bf5e-f0fb7d41db60") : configmap "kube-rbac-proxy" not found
Sep 15 07:22:32.118442 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:32.118226    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:32.118442 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:32.118291    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4adb8650-815d-427e-bf5e-f0fb7d41db60-insights-runtime-extractor-tls podName:4adb8650-815d-427e-bf5e-f0fb7d41db60 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:33.118278329 +0000 UTC m=+257.664848412 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4adb8650-815d-427e-bf5e-f0fb7d41db60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qk9cv" (UID: "4adb8650-815d-427e-bf5e-f0fb7d41db60") : secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:32.129063 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:32.129040    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-5rzzs"
Sep 15 07:22:32.394908 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:32.394830    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-5rzzs"]
Sep 15 07:22:32.397582 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:22:32.397555    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod299e0037_3343_48d4_bb94_080df4472b59.slice/crio-34d6334d3d9239aad5b8bff8536605f60d39411be8f8ab5e6c4ddc56e4d607c6 WatchSource:0}: Error finding container 34d6334d3d9239aad5b8bff8536605f60d39411be8f8ab5e6c4ddc56e4d607c6: Status 404 returned error can't find the container with id 34d6334d3d9239aad5b8bff8536605f60d39411be8f8ab5e6c4ddc56e4d607c6
Sep 15 07:22:32.703336 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:32.703305    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-5rzzs" event={"ID":"299e0037-3343-48d4-bb94-080df4472b59","Type":"ContainerStarted","Data":"34d6334d3d9239aad5b8bff8536605f60d39411be8f8ab5e6c4ddc56e4d607c6"}
Sep 15 07:22:33.126162 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:33.126073    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4adb8650-815d-427e-bf5e-f0fb7d41db60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:33.126162 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:33.126131    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4adb8650-815d-427e-bf5e-f0fb7d41db60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:33.126643 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:33.126204    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 07:22:33.126643 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:33.126304    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:33.126643 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:33.126305    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4adb8650-815d-427e-bf5e-f0fb7d41db60-kube-rbac-proxy-cm podName:4adb8650-815d-427e-bf5e-f0fb7d41db60 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:35.126286685 +0000 UTC m=+259.672856762 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4adb8650-815d-427e-bf5e-f0fb7d41db60-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qk9cv" (UID: "4adb8650-815d-427e-bf5e-f0fb7d41db60") : configmap "kube-rbac-proxy" not found
Sep 15 07:22:33.126643 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:33.126346    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4adb8650-815d-427e-bf5e-f0fb7d41db60-insights-runtime-extractor-tls podName:4adb8650-815d-427e-bf5e-f0fb7d41db60 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:35.126336085 +0000 UTC m=+259.672906175 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4adb8650-815d-427e-bf5e-f0fb7d41db60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qk9cv" (UID: "4adb8650-815d-427e-bf5e-f0fb7d41db60") : secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:33.570235 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:33.570207    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zvc79_a1932db5-d903-4b17-ac25-36613aa04ba4/dns-node-resolver/0.log"
Sep 15 07:22:33.707845 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:33.707808    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-5rzzs" event={"ID":"299e0037-3343-48d4-bb94-080df4472b59","Type":"ContainerStarted","Data":"e8be6a29dae0fc0d833f658ea67a68d7d7608c29ddc5e790778d5660bd08ad66"}
Sep 15 07:22:33.707988 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:33.707885    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-5rzzs" event={"ID":"299e0037-3343-48d4-bb94-080df4472b59","Type":"ContainerStarted","Data":"95bf279ab6d86f018b559c3b2319a3e35807156bdc198d3fe4918a9c12957ff6"}
Sep 15 07:22:33.723145 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:33.723090    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-5rzzs" podStartSLOduration=1.547435484 podStartE2EDuration="2.723074997s" totalImagesPullingTime="1.175639513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:22:33.722371764 +0000 UTC m=+258.268941879" watchObservedRunningTime="2026-09-15 07:22:33.723074997 +0000 UTC m=+258.269645097"
Sep 15 07:22:34.169673 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:34.169645    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4scf4_c38076ad-5b5b-4fff-8ad9-149153f9d9d2/node-ca/0.log"
Sep 15 07:22:34.840649 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:34.840610    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f86f2096-0758-472d-a436-90432eb7368c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-v6vxp\" (UID: \"f86f2096-0758-472d-a436-90432eb7368c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp"
Sep 15 07:22:34.840805 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:34.840726    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 07:22:34.840805 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:34.840777    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f86f2096-0758-472d-a436-90432eb7368c-cluster-monitoring-operator-tls podName:f86f2096-0758-472d-a436-90432eb7368c nodeName:}" failed. No retries permitted until 2026-09-15 07:22:42.840764114 +0000 UTC m=+267.387334204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f86f2096-0758-472d-a436-90432eb7368c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-v6vxp" (UID: "f86f2096-0758-472d-a436-90432eb7368c") : secret "cluster-monitoring-operator-tls" not found
Sep 15 07:22:34.941545 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:34.941506    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/591b847d-1325-4654-aaa1-e2d142921b00-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c85gj\" (UID: \"591b847d-1325-4654-aaa1-e2d142921b00\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c85gj"
Sep 15 07:22:34.941677 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:34.941604    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 07:22:34.941731 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:34.941689    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/591b847d-1325-4654-aaa1-e2d142921b00-samples-operator-tls podName:591b847d-1325-4654-aaa1-e2d142921b00 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:42.941668746 +0000 UTC m=+267.488238839 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/591b847d-1325-4654-aaa1-e2d142921b00-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-c85gj" (UID: "591b847d-1325-4654-aaa1-e2d142921b00") : secret "samples-operator-tls" not found
Sep 15 07:22:34.970657 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:34.970630    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5ltrc_1934978a-de53-4efa-8cb8-2f2bc312ceaf/serve-healthcheck-canary/0.log"
Sep 15 07:22:35.143807 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:35.143722    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4adb8650-815d-427e-bf5e-f0fb7d41db60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:35.143807 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:35.143765    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4adb8650-815d-427e-bf5e-f0fb7d41db60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:35.144000 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:35.143843    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 07:22:35.144000 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:35.143894    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:35.144000 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:35.143916    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4adb8650-815d-427e-bf5e-f0fb7d41db60-kube-rbac-proxy-cm podName:4adb8650-815d-427e-bf5e-f0fb7d41db60 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:39.143899606 +0000 UTC m=+263.690469684 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4adb8650-815d-427e-bf5e-f0fb7d41db60-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qk9cv" (UID: "4adb8650-815d-427e-bf5e-f0fb7d41db60") : configmap "kube-rbac-proxy" not found
Sep 15 07:22:35.144000 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:35.143936    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4adb8650-815d-427e-bf5e-f0fb7d41db60-insights-runtime-extractor-tls podName:4adb8650-815d-427e-bf5e-f0fb7d41db60 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:39.143923549 +0000 UTC m=+263.690493627 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4adb8650-815d-427e-bf5e-f0fb7d41db60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qk9cv" (UID: "4adb8650-815d-427e-bf5e-f0fb7d41db60") : secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:39.176035 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:39.175930    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4adb8650-815d-427e-bf5e-f0fb7d41db60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:39.176035 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:39.176020    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4adb8650-815d-427e-bf5e-f0fb7d41db60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:39.176488 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:39.176050    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 07:22:39.176488 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:39.176114    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4adb8650-815d-427e-bf5e-f0fb7d41db60-kube-rbac-proxy-cm podName:4adb8650-815d-427e-bf5e-f0fb7d41db60 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:47.176098166 +0000 UTC m=+271.722668259 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4adb8650-815d-427e-bf5e-f0fb7d41db60-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qk9cv" (UID: "4adb8650-815d-427e-bf5e-f0fb7d41db60") : configmap "kube-rbac-proxy" not found
Sep 15 07:22:39.176488 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:39.176170    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:39.176488 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:39.176225    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4adb8650-815d-427e-bf5e-f0fb7d41db60-insights-runtime-extractor-tls podName:4adb8650-815d-427e-bf5e-f0fb7d41db60 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:47.176211948 +0000 UTC m=+271.722782026 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4adb8650-815d-427e-bf5e-f0fb7d41db60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qk9cv" (UID: "4adb8650-815d-427e-bf5e-f0fb7d41db60") : secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:42.907840 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:42.907795    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f86f2096-0758-472d-a436-90432eb7368c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-v6vxp\" (UID: \"f86f2096-0758-472d-a436-90432eb7368c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp"
Sep 15 07:22:42.908245 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:42.907943    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 07:22:42.908245 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:42.908005    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f86f2096-0758-472d-a436-90432eb7368c-cluster-monitoring-operator-tls podName:f86f2096-0758-472d-a436-90432eb7368c nodeName:}" failed. No retries permitted until 2026-09-15 07:22:58.907986761 +0000 UTC m=+283.454556858 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f86f2096-0758-472d-a436-90432eb7368c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-v6vxp" (UID: "f86f2096-0758-472d-a436-90432eb7368c") : secret "cluster-monitoring-operator-tls" not found
Sep 15 07:22:43.008803 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:43.008763    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/591b847d-1325-4654-aaa1-e2d142921b00-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c85gj\" (UID: \"591b847d-1325-4654-aaa1-e2d142921b00\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c85gj"
Sep 15 07:22:43.011122 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:43.011101    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/591b847d-1325-4654-aaa1-e2d142921b00-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c85gj\" (UID: \"591b847d-1325-4654-aaa1-e2d142921b00\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c85gj"
Sep 15 07:22:43.298285 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:43.298241    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c85gj"
Sep 15 07:22:43.562212 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:43.562141    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c85gj"]
Sep 15 07:22:43.736997 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:43.736962    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c85gj" event={"ID":"591b847d-1325-4654-aaa1-e2d142921b00","Type":"ContainerStarted","Data":"7a6211bf42298b49646c8f184de1c273ed9a88162f6ca4a5834b1573e6ee1f6b"}
Sep 15 07:22:45.747564 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:45.747532    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c85gj" event={"ID":"591b847d-1325-4654-aaa1-e2d142921b00","Type":"ContainerStarted","Data":"fc29cd72a41dfa2eba82b4a4fe09b56417a3a5752eeb7f337c1a3ff40ecb8c69"}
Sep 15 07:22:45.748000 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:45.747583    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c85gj" event={"ID":"591b847d-1325-4654-aaa1-e2d142921b00","Type":"ContainerStarted","Data":"94d893ca36b0326ebf039c647daf34c708bf4eb92d409bc18d612032a510b924"}
Sep 15 07:22:45.764454 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:45.764410    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c85gj" podStartSLOduration=16.933029643 podStartE2EDuration="18.764392862s" totalImagesPullingTime="1.831363219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:22:45.763539591 +0000 UTC m=+270.310109694" watchObservedRunningTime="2026-09-15 07:22:45.764392862 +0000 UTC m=+270.310962958"
Sep 15 07:22:47.240760 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:47.240722    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4adb8650-815d-427e-bf5e-f0fb7d41db60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:47.240760 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:47.240769    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4adb8650-815d-427e-bf5e-f0fb7d41db60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:47.241208 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:47.240900    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:47.241208 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:47.240954    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4adb8650-815d-427e-bf5e-f0fb7d41db60-insights-runtime-extractor-tls podName:4adb8650-815d-427e-bf5e-f0fb7d41db60 nodeName:}" failed. No retries permitted until 2026-09-15 07:23:03.240940861 +0000 UTC m=+287.787510939 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4adb8650-815d-427e-bf5e-f0fb7d41db60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qk9cv" (UID: "4adb8650-815d-427e-bf5e-f0fb7d41db60") : secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:47.241324 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:47.241285    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4adb8650-815d-427e-bf5e-f0fb7d41db60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:22:56.458151 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:56.458110    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x" podUID="6a8acf07-ffd3-4101-a1d5-16770855954c"
Sep 15 07:22:56.458151 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:56.458117    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4" podUID="b8697c4f-de34-4c3f-ae67-fbbb429010c0"
Sep 15 07:22:56.458151 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:22:56.458110    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-qvk9j" podUID="64a7e365-d705-4174-992f-bd271f4b418e"
Sep 15 07:22:56.781464 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:56.781386    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:22:56.781598 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:56.781389    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:22:56.781598 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:56.781389    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x"
Sep 15 07:22:58.932974 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:58.932924    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f86f2096-0758-472d-a436-90432eb7368c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-v6vxp\" (UID: \"f86f2096-0758-472d-a436-90432eb7368c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp"
Sep 15 07:22:58.935285 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:58.935249    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f86f2096-0758-472d-a436-90432eb7368c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-v6vxp\" (UID: \"f86f2096-0758-472d-a436-90432eb7368c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp"
Sep 15 07:22:59.104116 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:59.104080    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp"
Sep 15 07:22:59.410125 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:59.410101    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp"]
Sep 15 07:22:59.739154 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:59.739115    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-wdl2x\" (UID: \"6a8acf07-ffd3-4101-a1d5-16770855954c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x"
Sep 15 07:22:59.739373 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:59.739180    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:22:59.741548 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:59.741518    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls\") pod \"image-registry-5bd85df4cd-n9nj4\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") " pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:22:59.741649 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:59.741628    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a8acf07-ffd3-4101-a1d5-16770855954c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-wdl2x\" (UID: \"6a8acf07-ffd3-4101-a1d5-16770855954c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x"
Sep 15 07:22:59.785362 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:59.785333    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rxml2\""
Sep 15 07:22:59.785362 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:59.785332    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nsfj7\""
Sep 15 07:22:59.791641 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:59.791612    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp" event={"ID":"f86f2096-0758-472d-a436-90432eb7368c","Type":"ContainerStarted","Data":"e9072b981bc73b118d05647f96c63cd7af96fafa3b37cf288a6498e303ceaaa8"}
Sep 15 07:22:59.792721 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:59.792711    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x"
Sep 15 07:22:59.792817 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:59.792803    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:22:59.840191 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:59.840167    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls\") pod \"dns-default-qvk9j\" (UID: \"64a7e365-d705-4174-992f-bd271f4b418e\") " pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:22:59.842538 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:22:59.842514    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64a7e365-d705-4174-992f-bd271f4b418e-metrics-tls\") pod \"dns-default-qvk9j\" (UID: \"64a7e365-d705-4174-992f-bd271f4b418e\") " pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:23:00.085531 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:00.085448    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tfj9f\""
Sep 15 07:23:00.092195 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:00.092163    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x"]
Sep 15 07:23:00.093654 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:00.093625    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:23:00.096290 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:23:00.096249    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a8acf07_ffd3_4101_a1d5_16770855954c.slice/crio-dbd81bc946e183d9245af933b817f400a7113c4b3f6e7c40b5256ca939e313aa WatchSource:0}: Error finding container dbd81bc946e183d9245af933b817f400a7113c4b3f6e7c40b5256ca939e313aa: Status 404 returned error can't find the container with id dbd81bc946e183d9245af933b817f400a7113c4b3f6e7c40b5256ca939e313aa
Sep 15 07:23:00.120179 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:00.120148    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bd85df4cd-n9nj4"]
Sep 15 07:23:00.121637 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:23:00.121612    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8697c4f_de34_4c3f_ae67_fbbb429010c0.slice/crio-ebfe2023e520ffd254fb27dc0473b5ceca6db0626f67c056d59951d9d5246799 WatchSource:0}: Error finding container ebfe2023e520ffd254fb27dc0473b5ceca6db0626f67c056d59951d9d5246799: Status 404 returned error can't find the container with id ebfe2023e520ffd254fb27dc0473b5ceca6db0626f67c056d59951d9d5246799
Sep 15 07:23:00.376803 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:00.376717    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qvk9j"]
Sep 15 07:23:00.379777 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:23:00.379749    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a7e365_d705_4174_992f_bd271f4b418e.slice/crio-7e1214cdac517f684d8871286b7dafec9b65bff1648f7ffb56ee6dbef4216f5e WatchSource:0}: Error finding container 7e1214cdac517f684d8871286b7dafec9b65bff1648f7ffb56ee6dbef4216f5e: Status 404 returned error can't find the container with id 7e1214cdac517f684d8871286b7dafec9b65bff1648f7ffb56ee6dbef4216f5e
Sep 15 07:23:00.796739 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:00.796702    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4" event={"ID":"b8697c4f-de34-4c3f-ae67-fbbb429010c0","Type":"ContainerStarted","Data":"c85b12c569fac539e1224c1de2b9a6285ff2159de01c30991d384b380ff046d0"}
Sep 15 07:23:00.796943 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:00.796762    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4" event={"ID":"b8697c4f-de34-4c3f-ae67-fbbb429010c0","Type":"ContainerStarted","Data":"ebfe2023e520ffd254fb27dc0473b5ceca6db0626f67c056d59951d9d5246799"}
Sep 15 07:23:00.796943 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:00.796792    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:23:00.797941 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:00.797916    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qvk9j" event={"ID":"64a7e365-d705-4174-992f-bd271f4b418e","Type":"ContainerStarted","Data":"7e1214cdac517f684d8871286b7dafec9b65bff1648f7ffb56ee6dbef4216f5e"}
Sep 15 07:23:00.799287 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:00.799247    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x" event={"ID":"6a8acf07-ffd3-4101-a1d5-16770855954c","Type":"ContainerStarted","Data":"dbd81bc946e183d9245af933b817f400a7113c4b3f6e7c40b5256ca939e313aa"}
Sep 15 07:23:00.816871 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:00.816809    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4" podStartSLOduration=284.816793451 podStartE2EDuration="4m44.816793451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:23:00.815582527 +0000 UTC m=+285.362152626" watchObservedRunningTime="2026-09-15 07:23:00.816793451 +0000 UTC m=+285.363363550"
Sep 15 07:23:01.804895 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:01.804828    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x" event={"ID":"6a8acf07-ffd3-4101-a1d5-16770855954c","Type":"ContainerStarted","Data":"6b34f468366c071ec905232b6d9229de98b281ecafff0ce36ab08e91c0ac2030"}
Sep 15 07:23:01.806911 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:01.806880    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp" event={"ID":"f86f2096-0758-472d-a436-90432eb7368c","Type":"ContainerStarted","Data":"0c9ebcf6a4c9f7389ac3c5c61a6ea34ac6ba939ec3dce2e2b904d18dd75b5674"}
Sep 15 07:23:01.823784 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:01.822650    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wdl2x" podStartSLOduration=268.255816237 podStartE2EDuration="4m29.822626799s" totalImagesPullingTime="1.566810562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:23:01.821461033 +0000 UTC m=+286.368031135" watchObservedRunningTime="2026-09-15 07:23:01.822626799 +0000 UTC m=+286.369196900"
Sep 15 07:23:01.840893 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:01.840820    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-v6vxp" podStartSLOduration=33.586485175 podStartE2EDuration="35.84079557s" totalImagesPullingTime="2.254310395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:23:01.840466432 +0000 UTC m=+286.387036558" watchObservedRunningTime="2026-09-15 07:23:01.84079557 +0000 UTC m=+286.387365669"
Sep 15 07:23:02.812928 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:02.812891    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qvk9j" event={"ID":"64a7e365-d705-4174-992f-bd271f4b418e","Type":"ContainerStarted","Data":"d8774061616522669994fef9cfab76a99f432a189a5631bb29412eb815827725"}
Sep 15 07:23:02.813418 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:02.812939    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qvk9j" event={"ID":"64a7e365-d705-4174-992f-bd271f4b418e","Type":"ContainerStarted","Data":"97c945cd9d9afa845c6af9c5cc034c99069ef355b94be713fc2d891dc56684a6"}
Sep 15 07:23:02.830761 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:02.830703    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qvk9j" podStartSLOduration=251.92317969 podStartE2EDuration="4m13.830667873s" totalImagesPullingTime="1.907488183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:23:02.829027986 +0000 UTC m=+287.375598106" watchObservedRunningTime="2026-09-15 07:23:02.830667873 +0000 UTC m=+287.377237971"
Sep 15 07:23:03.270289 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:03.270220    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4adb8650-815d-427e-bf5e-f0fb7d41db60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:23:03.272494 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:03.272473    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4adb8650-815d-427e-bf5e-f0fb7d41db60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qk9cv\" (UID: \"4adb8650-815d-427e-bf5e-f0fb7d41db60\") " pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:23:03.524743 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:03.524650    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qk9cv"
Sep 15 07:23:03.794625 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:03.792075    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qk9cv"]
Sep 15 07:23:03.817476 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:03.817451    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qk9cv" event={"ID":"4adb8650-815d-427e-bf5e-f0fb7d41db60","Type":"ContainerStarted","Data":"e1dc4b9ead1c57b5207e928c0eb5f8da4f2f580a7d560b3b5d02eff3f20858f7"}
Sep 15 07:23:03.817786 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:03.817578    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:23:04.824597 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:04.824513    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qk9cv" event={"ID":"4adb8650-815d-427e-bf5e-f0fb7d41db60","Type":"ContainerStarted","Data":"7f647e4e18161af0dc9e800f7efd9c0378b1b27a2746ba97c2ba19b79dafff40"}
Sep 15 07:23:04.824597 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:04.824568    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qk9cv" event={"ID":"4adb8650-815d-427e-bf5e-f0fb7d41db60","Type":"ContainerStarted","Data":"df461d75da7755cca3af56c08c753f0a07bec1b95e870752929ae141baa32573"}
Sep 15 07:23:08.906049 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:23:08.906007    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-jhh6r" podUID="bf0b2425-be3b-4095-b8ab-a44f426176ac"
Sep 15 07:23:09.849031 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:09.848929    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qk9cv" event={"ID":"4adb8650-815d-427e-bf5e-f0fb7d41db60","Type":"ContainerStarted","Data":"457aca49e29e89ae18443900c00f5372579fa0bb0759c77944ebb6bf946c7516"}
Sep 15 07:23:09.870473 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:09.870425    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qk9cv" podStartSLOduration=33.697199355 podStartE2EDuration="38.870411451s" totalImagesPullingTime="5.173212096s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:22:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:23:09.869589553 +0000 UTC m=+294.416159676" watchObservedRunningTime="2026-09-15 07:23:09.870411451 +0000 UTC m=+294.416981549"
Sep 15 07:23:13.826857 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:13.826827    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qvk9j"
Sep 15 07:23:14.586184 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.586151    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-b7cpz"]
Sep 15 07:23:14.589339 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.589318    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-b7cpz"
Sep 15 07:23:14.592857 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.592824    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-7cqcp\""
Sep 15 07:23:14.594090 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.594067    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 15 07:23:14.594193 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.594072    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 07:23:14.594193 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.594103    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 15 07:23:14.602760 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.602743    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"]
Sep 15 07:23:14.605811 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.605794    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"
Sep 15 07:23:14.606877 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.606860    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-b7cpz"]
Sep 15 07:23:14.609843 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.609825    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 15 07:23:14.610249 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.610227    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 15 07:23:14.610360 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.610284    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jppfx\""
Sep 15 07:23:14.610360 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.610331    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 15 07:23:14.618129 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.614989    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wr8bg"]
Sep 15 07:23:14.620101 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.620081    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.626441 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.626426    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 07:23:14.628231 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.628218    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9ch4s\""
Sep 15 07:23:14.628528 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.628514    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 07:23:14.628817 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.628803    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 07:23:14.633968 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.633945    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"]
Sep 15 07:23:14.754910 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.754875    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.754910 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.754912    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0dbc5764-bcfa-4bfd-8e85-dc6430bb3064-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-jdzrb\" (UID: \"0dbc5764-bcfa-4bfd-8e85-dc6430bb3064\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"
Sep 15 07:23:14.755103 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.754930    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0dbc5764-bcfa-4bfd-8e85-dc6430bb3064-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-jdzrb\" (UID: \"0dbc5764-bcfa-4bfd-8e85-dc6430bb3064\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"
Sep 15 07:23:14.755103 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.754981    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad63d264-6111-4b9c-be64-99e35ef4c05a-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-b7cpz\" (UID: \"ad63d264-6111-4b9c-be64-99e35ef4c05a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-b7cpz"
Sep 15 07:23:14.755103 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.755046    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dbc5764-bcfa-4bfd-8e85-dc6430bb3064-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-jdzrb\" (UID: \"0dbc5764-bcfa-4bfd-8e85-dc6430bb3064\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"
Sep 15 07:23:14.755103 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.755074    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf6bs\" (UniqueName: \"kubernetes.io/projected/ad63d264-6111-4b9c-be64-99e35ef4c05a-kube-api-access-xf6bs\") pod \"openshift-state-metrics-67dcf8c979-b7cpz\" (UID: \"ad63d264-6111-4b9c-be64-99e35ef4c05a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-b7cpz"
Sep 15 07:23:14.755103 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.755096    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0dbc5764-bcfa-4bfd-8e85-dc6430bb3064-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-jdzrb\" (UID: \"0dbc5764-bcfa-4bfd-8e85-dc6430bb3064\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"
Sep 15 07:23:14.755282 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.755122    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhkpx\" (UniqueName: \"kubernetes.io/projected/0dbc5764-bcfa-4bfd-8e85-dc6430bb3064-kube-api-access-dhkpx\") pod \"kube-state-metrics-59fbdd77d-jdzrb\" (UID: \"0dbc5764-bcfa-4bfd-8e85-dc6430bb3064\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"
Sep 15 07:23:14.755282 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.755139    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-root\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.755282 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.755161    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-node-exporter-textfile\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.755282 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.755180    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-node-exporter-wtmp\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.755282 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.755217    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-sys\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.755282 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.755234    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc5764-bcfa-4bfd-8e85-dc6430bb3064-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-jdzrb\" (UID: \"0dbc5764-bcfa-4bfd-8e85-dc6430bb3064\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"
Sep 15 07:23:14.755282 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.755248    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-node-exporter-tls\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.755568 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.755289    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-node-exporter-accelerators-collector-config\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.755568 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.755307    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96964\" (UniqueName: \"kubernetes.io/projected/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-kube-api-access-96964\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.755568 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.755332    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad63d264-6111-4b9c-be64-99e35ef4c05a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-b7cpz\" (UID: \"ad63d264-6111-4b9c-be64-99e35ef4c05a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-b7cpz"
Sep 15 07:23:14.755568 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.755419    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-metrics-client-ca\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.755568 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.755461    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad63d264-6111-4b9c-be64-99e35ef4c05a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-b7cpz\" (UID: \"ad63d264-6111-4b9c-be64-99e35ef4c05a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-b7cpz"
Sep 15 07:23:14.856680 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.856606    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.856680 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.856649    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0dbc5764-bcfa-4bfd-8e85-dc6430bb3064-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-jdzrb\" (UID: \"0dbc5764-bcfa-4bfd-8e85-dc6430bb3064\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"
Sep 15 07:23:14.856680 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.856667    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0dbc5764-bcfa-4bfd-8e85-dc6430bb3064-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-jdzrb\" (UID: \"0dbc5764-bcfa-4bfd-8e85-dc6430bb3064\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"
Sep 15 07:23:14.857223 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.856687    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad63d264-6111-4b9c-be64-99e35ef4c05a-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-b7cpz\" (UID: \"ad63d264-6111-4b9c-be64-99e35ef4c05a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-b7cpz"
Sep 15 07:23:14.857223 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.856740    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dbc5764-bcfa-4bfd-8e85-dc6430bb3064-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-jdzrb\" (UID: \"0dbc5764-bcfa-4bfd-8e85-dc6430bb3064\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"
Sep 15 07:23:14.857223 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.856767    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf6bs\" (UniqueName: \"kubernetes.io/projected/ad63d264-6111-4b9c-be64-99e35ef4c05a-kube-api-access-xf6bs\") pod \"openshift-state-metrics-67dcf8c979-b7cpz\" (UID: \"ad63d264-6111-4b9c-be64-99e35ef4c05a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-b7cpz"
Sep 15 07:23:14.857223 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.856788    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0dbc5764-bcfa-4bfd-8e85-dc6430bb3064-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-jdzrb\" (UID: \"0dbc5764-bcfa-4bfd-8e85-dc6430bb3064\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"
Sep 15 07:23:14.857223 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.856823    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhkpx\" (UniqueName: \"kubernetes.io/projected/0dbc5764-bcfa-4bfd-8e85-dc6430bb3064-kube-api-access-dhkpx\") pod \"kube-state-metrics-59fbdd77d-jdzrb\" (UID: \"0dbc5764-bcfa-4bfd-8e85-dc6430bb3064\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"
Sep 15 07:23:14.857223 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.856842    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-root\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.857223 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.856867    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-node-exporter-textfile\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.857223 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.856891    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-node-exporter-wtmp\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.857223 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.856917    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-sys\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.857223 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.856941    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc5764-bcfa-4bfd-8e85-dc6430bb3064-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-jdzrb\" (UID: \"0dbc5764-bcfa-4bfd-8e85-dc6430bb3064\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"
Sep 15 07:23:14.857223 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.856961    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-node-exporter-tls\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.857223 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.856982    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-node-exporter-accelerators-collector-config\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.857223 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.857011    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96964\" (UniqueName: \"kubernetes.io/projected/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-kube-api-access-96964\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.857223 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.857045    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad63d264-6111-4b9c-be64-99e35ef4c05a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-b7cpz\" (UID: \"ad63d264-6111-4b9c-be64-99e35ef4c05a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-b7cpz"
Sep 15 07:23:14.857223 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.857080    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-metrics-client-ca\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.857223 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.857084    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0dbc5764-bcfa-4bfd-8e85-dc6430bb3064-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-jdzrb\" (UID: \"0dbc5764-bcfa-4bfd-8e85-dc6430bb3064\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"
Sep 15 07:23:14.858051 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.857112    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad63d264-6111-4b9c-be64-99e35ef4c05a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-b7cpz\" (UID: \"ad63d264-6111-4b9c-be64-99e35ef4c05a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-b7cpz"
Sep 15 07:23:14.858051 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.857174    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-root\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.858051 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.857229    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-sys\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.858051 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.857383    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-node-exporter-textfile\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.858051 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.857590    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad63d264-6111-4b9c-be64-99e35ef4c05a-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-b7cpz\" (UID: \"ad63d264-6111-4b9c-be64-99e35ef4c05a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-b7cpz"
Sep 15 07:23:14.858330 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.858071    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0dbc5764-bcfa-4bfd-8e85-dc6430bb3064-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-jdzrb\" (UID: \"0dbc5764-bcfa-4bfd-8e85-dc6430bb3064\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"
Sep 15 07:23:14.858330 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.858173    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc5764-bcfa-4bfd-8e85-dc6430bb3064-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-jdzrb\" (UID: \"0dbc5764-bcfa-4bfd-8e85-dc6430bb3064\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"
Sep 15 07:23:14.858830 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.858801    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-metrics-client-ca\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.859002 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.858850    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-node-exporter-accelerators-collector-config\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.859002 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.858870    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-node-exporter-wtmp\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.859593 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.859546    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.860293 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.860271    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad63d264-6111-4b9c-be64-99e35ef4c05a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-b7cpz\" (UID: \"ad63d264-6111-4b9c-be64-99e35ef4c05a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-b7cpz"
Sep 15 07:23:14.860569 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.860546    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-node-exporter-tls\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.860647 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.860570    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0dbc5764-bcfa-4bfd-8e85-dc6430bb3064-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-jdzrb\" (UID: \"0dbc5764-bcfa-4bfd-8e85-dc6430bb3064\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"
Sep 15 07:23:14.861343 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.861323    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad63d264-6111-4b9c-be64-99e35ef4c05a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-b7cpz\" (UID: \"ad63d264-6111-4b9c-be64-99e35ef4c05a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-b7cpz"
Sep 15 07:23:14.861516 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.861495    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dbc5764-bcfa-4bfd-8e85-dc6430bb3064-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-jdzrb\" (UID: \"0dbc5764-bcfa-4bfd-8e85-dc6430bb3064\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"
Sep 15 07:23:14.865900 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.865877    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhkpx\" (UniqueName: \"kubernetes.io/projected/0dbc5764-bcfa-4bfd-8e85-dc6430bb3064-kube-api-access-dhkpx\") pod \"kube-state-metrics-59fbdd77d-jdzrb\" (UID: \"0dbc5764-bcfa-4bfd-8e85-dc6430bb3064\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"
Sep 15 07:23:14.866336 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.866317    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96964\" (UniqueName: \"kubernetes.io/projected/7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc-kube-api-access-96964\") pod \"node-exporter-wr8bg\" (UID: \"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc\") " pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.866528 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.866511    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf6bs\" (UniqueName: \"kubernetes.io/projected/ad63d264-6111-4b9c-be64-99e35ef4c05a-kube-api-access-xf6bs\") pod \"openshift-state-metrics-67dcf8c979-b7cpz\" (UID: \"ad63d264-6111-4b9c-be64-99e35ef4c05a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-b7cpz"
Sep 15 07:23:14.898420 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.898394    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-b7cpz"
Sep 15 07:23:14.914576 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.914552    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"
Sep 15 07:23:14.928324 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:14.928300    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wr8bg"
Sep 15 07:23:14.939148 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:23:14.939125    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6cb60b_f7af_49bb_8a3e_70f1ce0c0dfc.slice/crio-dbef75a26afa055c312530dc076e10781712155e3ec4122090d12eaa0ccae479 WatchSource:0}: Error finding container dbef75a26afa055c312530dc076e10781712155e3ec4122090d12eaa0ccae479: Status 404 returned error can't find the container with id dbef75a26afa055c312530dc076e10781712155e3ec4122090d12eaa0ccae479
Sep 15 07:23:15.175495 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:15.175429    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-b7cpz"]
Sep 15 07:23:15.177460 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:23:15.177428    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad63d264_6111_4b9c_be64_99e35ef4c05a.slice/crio-98dc3a57a419796661da7528183c37d695004fbc5687c9a9786fdc0e52fb84b5 WatchSource:0}: Error finding container 98dc3a57a419796661da7528183c37d695004fbc5687c9a9786fdc0e52fb84b5: Status 404 returned error can't find the container with id 98dc3a57a419796661da7528183c37d695004fbc5687c9a9786fdc0e52fb84b5
Sep 15 07:23:15.200868 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:15.200844    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb"]
Sep 15 07:23:15.211334 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:23:15.211309    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbc5764_bcfa_4bfd_8e85_dc6430bb3064.slice/crio-9d3a649a9dbca313b31db960b5ee2b95179ef0f2029fe3564dcc7ca4295b345e WatchSource:0}: Error finding container 9d3a649a9dbca313b31db960b5ee2b95179ef0f2029fe3564dcc7ca4295b345e: Status 404 returned error can't find the container with id 9d3a649a9dbca313b31db960b5ee2b95179ef0f2029fe3564dcc7ca4295b345e
Sep 15 07:23:15.807687 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:15.807661    2571 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 07:23:15.874399 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:15.874046    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-b7cpz" event={"ID":"ad63d264-6111-4b9c-be64-99e35ef4c05a","Type":"ContainerStarted","Data":"4d1c5a3a4d02ccbf23783bc212cd09bde13cdd2864befac116090ea34ec0af0f"}
Sep 15 07:23:15.874399 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:15.874111    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-b7cpz" event={"ID":"ad63d264-6111-4b9c-be64-99e35ef4c05a","Type":"ContainerStarted","Data":"d043237af721cc914683147c6e4aded7792b070ca9f87cf85df0d437da4e4fd7"}
Sep 15 07:23:15.874399 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:15.874139    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-b7cpz" event={"ID":"ad63d264-6111-4b9c-be64-99e35ef4c05a","Type":"ContainerStarted","Data":"98dc3a57a419796661da7528183c37d695004fbc5687c9a9786fdc0e52fb84b5"}
Sep 15 07:23:15.876555 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:15.876484    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wr8bg" event={"ID":"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc","Type":"ContainerStarted","Data":"dbef75a26afa055c312530dc076e10781712155e3ec4122090d12eaa0ccae479"}
Sep 15 07:23:15.877930 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:15.877903    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb" event={"ID":"0dbc5764-bcfa-4bfd-8e85-dc6430bb3064","Type":"ContainerStarted","Data":"9d3a649a9dbca313b31db960b5ee2b95179ef0f2029fe3564dcc7ca4295b345e"}
Sep 15 07:23:16.594550 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.594515    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-67fdb78968-hf97j"]
Sep 15 07:23:16.597932 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.597914    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.600412 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.600389    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 15 07:23:16.600540 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.600420    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9o96gfcp40mgr\""
Sep 15 07:23:16.600540 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.600468    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 15 07:23:16.600540 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.600487    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7clvh\""
Sep 15 07:23:16.600540 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.600430    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 15 07:23:16.600540 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.600490    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 15 07:23:16.600783 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.600710    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 15 07:23:16.608877 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.608855    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67fdb78968-hf97j"]
Sep 15 07:23:16.775862 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.775826    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-secret-thanos-querier-tls\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.775981 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.775875    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.775981 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.775913    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.775981 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.775970    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-secret-grpc-tls\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.776145 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.775996    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.776145 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.776022    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-metrics-client-ca\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.776145 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.776062    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlvb2\" (UniqueName: \"kubernetes.io/projected/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-kube-api-access-dlvb2\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.776145 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.776100    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.877078 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.876991    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-secret-grpc-tls\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.877078 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.877031    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.877078 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.877051    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-metrics-client-ca\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.877997 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.877090    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlvb2\" (UniqueName: \"kubernetes.io/projected/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-kube-api-access-dlvb2\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.877997 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.877123    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.877997 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.877193    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-secret-thanos-querier-tls\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.877997 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.877219    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.877997 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.877281    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.877997 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.877932    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-metrics-client-ca\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.880316 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.880284    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.880442 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.880383    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-secret-thanos-querier-tls\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.880554 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.880532    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.880935 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.880892    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.881085 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.881061    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-secret-grpc-tls\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.881550 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.881525    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.886920 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.886884    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb" event={"ID":"0dbc5764-bcfa-4bfd-8e85-dc6430bb3064","Type":"ContainerStarted","Data":"2018a8cfa9ff8cd37e26591474fb7c0bfdb1698d9781d12244d60a117361ba38"}
Sep 15 07:23:16.887018 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.886959    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb" event={"ID":"0dbc5764-bcfa-4bfd-8e85-dc6430bb3064","Type":"ContainerStarted","Data":"122fbc4ba46b249418a54149193dd36ea7d276f0060251299ad504b09f474eea"}
Sep 15 07:23:16.887018 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.886987    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb" event={"ID":"0dbc5764-bcfa-4bfd-8e85-dc6430bb3064","Type":"ContainerStarted","Data":"322fb80637d0e42c4581fa9e597b73f83f983e637a465dbc7e70a8b7f466f816"}
Sep 15 07:23:16.887472 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.887449    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlvb2\" (UniqueName: \"kubernetes.io/projected/4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62-kube-api-access-dlvb2\") pod \"thanos-querier-67fdb78968-hf97j\" (UID: \"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62\") " pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.889309 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.889285    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-b7cpz" event={"ID":"ad63d264-6111-4b9c-be64-99e35ef4c05a","Type":"ContainerStarted","Data":"80d4f97114cedffcc674648e92cf1430e9057915482c9a67e69b42a1a21b0e38"}
Sep 15 07:23:16.904356 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.904311    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-jdzrb" podStartSLOduration=1.501949664 podStartE2EDuration="2.904297852s" totalImagesPullingTime="1.402348188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:23:16.903360125 +0000 UTC m=+301.449930238" watchObservedRunningTime="2026-09-15 07:23:16.904297852 +0000 UTC m=+301.450867952"
Sep 15 07:23:16.907235 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.907212    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:16.920609 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:16.920535    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-b7cpz" podStartSLOduration=1.596122474 podStartE2EDuration="2.920517921s" totalImagesPullingTime="1.324395447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:23:16.919137355 +0000 UTC m=+301.465707502" watchObservedRunningTime="2026-09-15 07:23:16.920517921 +0000 UTC m=+301.467088020"
Sep 15 07:23:17.190852 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:17.190773    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67fdb78968-hf97j"]
Sep 15 07:23:17.193947 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:23:17.193909    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2ec3c6_7c55_4e3e_bd10_ca26b10bfc62.slice/crio-61579dd58fe80fa368f3b7159b3bc4eff5a3697d06250f96302cce05931b1e90 WatchSource:0}: Error finding container 61579dd58fe80fa368f3b7159b3bc4eff5a3697d06250f96302cce05931b1e90: Status 404 returned error can't find the container with id 61579dd58fe80fa368f3b7159b3bc4eff5a3697d06250f96302cce05931b1e90
Sep 15 07:23:17.195957 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:17.195943    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 07:23:17.250031 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:17.249999    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bd85df4cd-n9nj4"]
Sep 15 07:23:17.254271 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:17.254230    2571 patch_prober.go:28] interesting pod/image-registry-5bd85df4cd-n9nj4 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 15 07:23:17.254388 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:17.254314    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4" podUID="b8697c4f-de34-4c3f-ae67-fbbb429010c0" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 07:23:17.893626 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:17.893578    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j" event={"ID":"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62","Type":"ContainerStarted","Data":"61579dd58fe80fa368f3b7159b3bc4eff5a3697d06250f96302cce05931b1e90"}
Sep 15 07:23:18.899345 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:18.899310    2571 generic.go:356] "Generic (PLEG): container finished" podID="7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc" containerID="dab45cb0829db7286261a224b25dc5cc3b9ae0d28826696ea3966e87664a2468" exitCode=0
Sep 15 07:23:18.899680 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:18.899374    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wr8bg" event={"ID":"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc","Type":"ContainerDied","Data":"dab45cb0829db7286261a224b25dc5cc3b9ae0d28826696ea3966e87664a2468"}
Sep 15 07:23:18.942573 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:18.942547    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-8b6949c-j2xf4"]
Sep 15 07:23:18.947985 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:18.947968    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:18.951178 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:18.951142    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 15 07:23:18.951304 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:18.951197    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 15 07:23:18.951386 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:18.951316    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kb6b8\""
Sep 15 07:23:18.952134 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:18.952114    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-98mf6tckncmcd\""
Sep 15 07:23:18.952293 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:18.952122    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 15 07:23:18.952404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:18.952387    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 15 07:23:18.965153 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:18.965131    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8b6949c-j2xf4"]
Sep 15 07:23:19.099242 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.099198    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8b6949c-j2xf4\" (UID: \"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9\") " pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:19.099439 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.099291    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmqpn\" (UniqueName: \"kubernetes.io/projected/0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9-kube-api-access-gmqpn\") pod \"metrics-server-8b6949c-j2xf4\" (UID: \"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9\") " pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:19.099439 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.099328    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9-secret-metrics-server-client-certs\") pod \"metrics-server-8b6949c-j2xf4\" (UID: \"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9\") " pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:19.099439 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.099368    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9-secret-metrics-server-tls\") pod \"metrics-server-8b6949c-j2xf4\" (UID: \"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9\") " pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:19.099439 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.099434    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9-audit-log\") pod \"metrics-server-8b6949c-j2xf4\" (UID: \"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9\") " pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:19.099599 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.099455    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9-metrics-server-audit-profiles\") pod \"metrics-server-8b6949c-j2xf4\" (UID: \"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9\") " pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:19.099599 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.099492    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9-client-ca-bundle\") pod \"metrics-server-8b6949c-j2xf4\" (UID: \"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9\") " pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:19.200583 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.200550    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9-client-ca-bundle\") pod \"metrics-server-8b6949c-j2xf4\" (UID: \"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9\") " pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:19.200753 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.200638    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8b6949c-j2xf4\" (UID: \"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9\") " pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:19.200753 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.200690    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmqpn\" (UniqueName: \"kubernetes.io/projected/0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9-kube-api-access-gmqpn\") pod \"metrics-server-8b6949c-j2xf4\" (UID: \"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9\") " pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:19.200753 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.200720    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9-secret-metrics-server-client-certs\") pod \"metrics-server-8b6949c-j2xf4\" (UID: \"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9\") " pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:19.200929 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.200752    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9-secret-metrics-server-tls\") pod \"metrics-server-8b6949c-j2xf4\" (UID: \"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9\") " pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:19.200929 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.200790    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9-audit-log\") pod \"metrics-server-8b6949c-j2xf4\" (UID: \"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9\") " pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:19.200929 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.200810    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9-metrics-server-audit-profiles\") pod \"metrics-server-8b6949c-j2xf4\" (UID: \"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9\") " pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:19.201613 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.201546    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8b6949c-j2xf4\" (UID: \"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9\") " pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:19.201836 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.201796    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9-metrics-server-audit-profiles\") pod \"metrics-server-8b6949c-j2xf4\" (UID: \"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9\") " pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:19.201971 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.201953    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9-audit-log\") pod \"metrics-server-8b6949c-j2xf4\" (UID: \"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9\") " pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:19.203721 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.203690    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9-secret-metrics-server-client-certs\") pod \"metrics-server-8b6949c-j2xf4\" (UID: \"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9\") " pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:19.203827 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.203723    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9-client-ca-bundle\") pod \"metrics-server-8b6949c-j2xf4\" (UID: \"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9\") " pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:19.203827 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.203787    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9-secret-metrics-server-tls\") pod \"metrics-server-8b6949c-j2xf4\" (UID: \"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9\") " pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:19.209233 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.209209    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmqpn\" (UniqueName: \"kubernetes.io/projected/0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9-kube-api-access-gmqpn\") pod \"metrics-server-8b6949c-j2xf4\" (UID: \"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9\") " pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:19.259496 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.259453    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:19.360166 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.360136    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-chnj2"]
Sep 15 07:23:19.363505 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.363472    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-chnj2"
Sep 15 07:23:19.365816 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.365794    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 15 07:23:19.365952 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.365802    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7k85l\""
Sep 15 07:23:19.375149 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.375103    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-chnj2"]
Sep 15 07:23:19.503619 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.503527    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7tsg\" (UniqueName: \"kubernetes.io/projected/d8499382-718b-41c0-a32e-ca7118b6efd0-kube-api-access-f7tsg\") pod \"monitoring-plugin-5d85778b9d-chnj2\" (UID: \"d8499382-718b-41c0-a32e-ca7118b6efd0\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-chnj2"
Sep 15 07:23:19.503758 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.503646    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d8499382-718b-41c0-a32e-ca7118b6efd0-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-chnj2\" (UID: \"d8499382-718b-41c0-a32e-ca7118b6efd0\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-chnj2"
Sep 15 07:23:19.604727 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.604699    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7tsg\" (UniqueName: \"kubernetes.io/projected/d8499382-718b-41c0-a32e-ca7118b6efd0-kube-api-access-f7tsg\") pod \"monitoring-plugin-5d85778b9d-chnj2\" (UID: \"d8499382-718b-41c0-a32e-ca7118b6efd0\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-chnj2"
Sep 15 07:23:19.604828 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.604813    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d8499382-718b-41c0-a32e-ca7118b6efd0-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-chnj2\" (UID: \"d8499382-718b-41c0-a32e-ca7118b6efd0\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-chnj2"
Sep 15 07:23:19.606988 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.606965    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d8499382-718b-41c0-a32e-ca7118b6efd0-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-chnj2\" (UID: \"d8499382-718b-41c0-a32e-ca7118b6efd0\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-chnj2"
Sep 15 07:23:19.613208 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.613185    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7tsg\" (UniqueName: \"kubernetes.io/projected/d8499382-718b-41c0-a32e-ca7118b6efd0-kube-api-access-f7tsg\") pod \"monitoring-plugin-5d85778b9d-chnj2\" (UID: \"d8499382-718b-41c0-a32e-ca7118b6efd0\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-chnj2"
Sep 15 07:23:19.676047 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.676024    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-chnj2"
Sep 15 07:23:19.813445 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.813312    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8b6949c-j2xf4"]
Sep 15 07:23:19.816358 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:23:19.816329    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b3c282e_4fc2_4c16_bfa5_cd7ba72a02d9.slice/crio-1767edabb767492a5b2fa0676e6cd1e9770abe924cbf19cf3b491a1f63047984 WatchSource:0}: Error finding container 1767edabb767492a5b2fa0676e6cd1e9770abe924cbf19cf3b491a1f63047984: Status 404 returned error can't find the container with id 1767edabb767492a5b2fa0676e6cd1e9770abe924cbf19cf3b491a1f63047984
Sep 15 07:23:19.909897 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.909859    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wr8bg" event={"ID":"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc","Type":"ContainerStarted","Data":"4c3e7d905db77c5544262b02d09629913a41f136467e555c2a95800c2a2c0663"}
Sep 15 07:23:19.910293 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.909914    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wr8bg" event={"ID":"7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc","Type":"ContainerStarted","Data":"c530e05ddc74016409a98d60821a536e3c84f9687b1397a8f5e1eb8d08a187f8"}
Sep 15 07:23:19.910293 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.910088    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j" event={"ID":"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62","Type":"ContainerStarted","Data":"d6c54022a7cf06349b5a3a69ca7dfc174840b1d3694e857778fb4bc001c3e04c"}
Sep 15 07:23:19.910293 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.910145    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j" event={"ID":"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62","Type":"ContainerStarted","Data":"d868f8ad2786e7187237170bda2b6f2704b90da997af6a451ad6ec46e8a68a91"}
Sep 15 07:23:19.910293 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.910170    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j" event={"ID":"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62","Type":"ContainerStarted","Data":"8210f46772040591937c2dd4d506139e9873c5cbd2b0ba5c3d7f77e1ae1e6fa8"}
Sep 15 07:23:19.911150 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.911132    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8b6949c-j2xf4" event={"ID":"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9","Type":"ContainerStarted","Data":"1767edabb767492a5b2fa0676e6cd1e9770abe924cbf19cf3b491a1f63047984"}
Sep 15 07:23:19.927702 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.927652    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wr8bg" podStartSLOduration=2.13523162 podStartE2EDuration="5.927636662s" totalImagesPullingTime="3.730950579s" totalInitContainerRuntime="61.454463ms" isStatefulPod=true podCreationTimestamp="2026-09-15 07:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:23:19.92612052 +0000 UTC m=+304.472690622" watchObservedRunningTime="2026-09-15 07:23:19.927636662 +0000 UTC m=+304.474206761"
Sep 15 07:23:19.950161 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:19.950135    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-chnj2"]
Sep 15 07:23:19.953328 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:23:19.953305    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8499382_718b_41c0_a32e_ca7118b6efd0.slice/crio-b18cac67c9b16e3c1948ec5a68d1f2bfe798b6af8fb4d96b6db0c031fb65d155 WatchSource:0}: Error finding container b18cac67c9b16e3c1948ec5a68d1f2bfe798b6af8fb4d96b6db0c031fb65d155: Status 404 returned error can't find the container with id b18cac67c9b16e3c1948ec5a68d1f2bfe798b6af8fb4d96b6db0c031fb65d155
Sep 15 07:23:20.771315 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.771243    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 07:23:20.776115 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.776005    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:20.779219 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.779170    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 07:23:20.779360 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.779235    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 07:23:20.779360 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.779290    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bkl0j842qou9r\""
Sep 15 07:23:20.779522 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.779501    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 07:23:20.779580 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.779554    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 07:23:20.779634 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.779582    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 07:23:20.780057 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.780030    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 07:23:20.780283 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.780247    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 07:23:20.781300 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.780478    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 07:23:20.781300 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.780767    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 07:23:20.781300 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.780907    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 07:23:20.781300 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.780943    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 07:23:20.781300 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.780953    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fd9vs\""
Sep 15 07:23:20.781601 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.781574    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 07:23:20.781905 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.781758    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 07:23:20.783368 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.783311    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 07:23:20.794404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.793898    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 07:23:20.916988 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.916954    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:20.917467 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.917033    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:20.917467 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.917069    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:20.917467 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.917098    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d996759f-03a2-4cba-804e-ab4ae2468df1-config-out\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:20.917467 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.917127    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-config\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:20.917467 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.917158    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:20.917467 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.917181    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:20.917467 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.917219    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d996759f-03a2-4cba-804e-ab4ae2468df1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:20.917467 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.917243    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:20.917467 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.917285    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-web-config\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:20.917467 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.917315    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjpb9\" (UniqueName: \"kubernetes.io/projected/d996759f-03a2-4cba-804e-ab4ae2468df1-kube-api-access-vjpb9\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:20.917467 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.917346    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:20.917467 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.917435    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:20.917977 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.917475    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:20.917977 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.917508    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:20.917977 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.917559    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:20.917977 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.917586    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:20.917977 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.917611    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:20.917977 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.917643    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:20.917977 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.917679    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:20.918326 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:20.918054    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-chnj2" event={"ID":"d8499382-718b-41c0-a32e-ca7118b6efd0","Type":"ContainerStarted","Data":"b18cac67c9b16e3c1948ec5a68d1f2bfe798b6af8fb4d96b6db0c031fb65d155"}
Sep 15 07:23:21.018951 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.018909    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.019133 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.018964    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d996759f-03a2-4cba-804e-ab4ae2468df1-config-out\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.019133 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.018993    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-config\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.019133 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.019018    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.019133 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.019044    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.022187 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.019532    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d996759f-03a2-4cba-804e-ab4ae2468df1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.022187 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.019577    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.022187 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.019628    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-web-config\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.022187 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.019669    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjpb9\" (UniqueName: \"kubernetes.io/projected/d996759f-03a2-4cba-804e-ab4ae2468df1-kube-api-access-vjpb9\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.022187 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.019713    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.022187 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.019751    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.022187 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.019755    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.022187 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.019810    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.022187 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.019854    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.022187 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.019890    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.022187 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.019914    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.022187 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.019944    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.022187 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.019974    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.022187 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.020029    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.022187 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.020034    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.022187 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.020071    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.022187 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.020135    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.023058 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.020348    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.023058 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.020396    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.023058 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.020960    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.023058 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.022841    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.023058 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.022882    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-config\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.023058 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.022952    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.023058 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.023009    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d996759f-03a2-4cba-804e-ab4ae2468df1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.024960 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.023908    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d996759f-03a2-4cba-804e-ab4ae2468df1-config-out\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.024960 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.024459    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.026106 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.026036    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.026194 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.026163    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.026538 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.026324    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.027050 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.027008    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.027612 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.027575    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.027612 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.027588    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-web-config\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.028176 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.028152    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.028542 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.028519    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.038581 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.038558    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjpb9\" (UniqueName: \"kubernetes.io/projected/d996759f-03a2-4cba-804e-ab4ae2468df1-kube-api-access-vjpb9\") pod \"prometheus-k8s-0\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.091415 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.091378    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:21.930435 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.930339    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j" event={"ID":"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62","Type":"ContainerStarted","Data":"e65fb94aea18150e21bb716dd44a7d785a5b08e84e987e8534ea9dca56c130fb"}
Sep 15 07:23:21.930881 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.930448    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j" event={"ID":"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62","Type":"ContainerStarted","Data":"511f7bc51ac95becc80e428053d0edc419b5f0716cd819edbac3d37b50aaede0"}
Sep 15 07:23:21.930881 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.930499    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:21.930881 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.930535    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j" event={"ID":"4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62","Type":"ContainerStarted","Data":"e7ebea1ff8520c265d2acf06f980244990ef6f47d45b8203d5c85476e0cf88f8"}
Sep 15 07:23:21.932086 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.932061    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8b6949c-j2xf4" event={"ID":"0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9","Type":"ContainerStarted","Data":"dfa5bb5251c10495e00560ddcb16c9291267a2208f341e0486f6e4ef1f9a3cd9"}
Sep 15 07:23:21.933696 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.933666    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-chnj2" event={"ID":"d8499382-718b-41c0-a32e-ca7118b6efd0","Type":"ContainerStarted","Data":"472639feaf356feb4fafdd32dff5907f29c44f2de1086087c46007a50805ccb8"}
Sep 15 07:23:21.933807 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.933793    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-chnj2"
Sep 15 07:23:21.938658 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.938642    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-chnj2"
Sep 15 07:23:21.953071 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.953050    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 07:23:21.955710 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:23:21.955687    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd996759f_03a2_4cba_804e_ab4ae2468df1.slice/crio-7250793c9a4d974a197e29866d6fe5594a7b99b9103d22347fa801be07de67c0 WatchSource:0}: Error finding container 7250793c9a4d974a197e29866d6fe5594a7b99b9103d22347fa801be07de67c0: Status 404 returned error can't find the container with id 7250793c9a4d974a197e29866d6fe5594a7b99b9103d22347fa801be07de67c0
Sep 15 07:23:21.958285 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.958209    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j" podStartSLOduration=1.747949975 podStartE2EDuration="5.958196155s" totalImagesPullingTime="4.21024618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:23:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:23:21.956496997 +0000 UTC m=+306.503067110" watchObservedRunningTime="2026-09-15 07:23:21.958196155 +0000 UTC m=+306.504766254"
Sep 15 07:23:21.973153 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.973116    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-chnj2" podStartSLOduration=1.255014538 podStartE2EDuration="2.973103271s" totalImagesPullingTime="1.718088733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:23:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:23:21.971384475 +0000 UTC m=+306.517954575" watchObservedRunningTime="2026-09-15 07:23:21.973103271 +0000 UTC m=+306.519673369"
Sep 15 07:23:21.990267 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:21.990197    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-8b6949c-j2xf4" podStartSLOduration=2.138595937 podStartE2EDuration="3.990177726s" totalImagesPullingTime="1.851581789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:23:21.989564521 +0000 UTC m=+306.536134660" watchObservedRunningTime="2026-09-15 07:23:21.990177726 +0000 UTC m=+306.536747827"
Sep 15 07:23:22.940165 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:22.940117    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d996759f-03a2-4cba-804e-ab4ae2468df1","Type":"ContainerStarted","Data":"7250793c9a4d974a197e29866d6fe5594a7b99b9103d22347fa801be07de67c0"}
Sep 15 07:23:23.903007 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:23.902970    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:23:23.945726 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:23.945681    2571 generic.go:356] "Generic (PLEG): container finished" podID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerID="58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61" exitCode=0
Sep 15 07:23:23.946081 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:23.945764    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d996759f-03a2-4cba-804e-ab4ae2468df1","Type":"ContainerDied","Data":"58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61"}
Sep 15 07:23:27.254520 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:27.254487    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:23:27.947003 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:27.946975    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-67fdb78968-hf97j"
Sep 15 07:23:30.977965 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:30.977925    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d996759f-03a2-4cba-804e-ab4ae2468df1","Type":"ContainerStarted","Data":"7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e"}
Sep 15 07:23:30.978342 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:30.977987    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d996759f-03a2-4cba-804e-ab4ae2468df1","Type":"ContainerStarted","Data":"917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880"}
Sep 15 07:23:30.978342 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:30.978013    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d996759f-03a2-4cba-804e-ab4ae2468df1","Type":"ContainerStarted","Data":"6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78"}
Sep 15 07:23:30.978342 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:30.978037    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d996759f-03a2-4cba-804e-ab4ae2468df1","Type":"ContainerStarted","Data":"0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb"}
Sep 15 07:23:30.978342 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:30.978064    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d996759f-03a2-4cba-804e-ab4ae2468df1","Type":"ContainerStarted","Data":"eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff"}
Sep 15 07:23:30.978342 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:30.978089    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d996759f-03a2-4cba-804e-ab4ae2468df1","Type":"ContainerStarted","Data":"4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f"}
Sep 15 07:23:31.006880 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:31.006826    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.934188477 podStartE2EDuration="11.006810962s" totalImagesPullingTime="7.050231965s" totalInitContainerRuntime="22.39052ms" isStatefulPod=false podCreationTimestamp="2026-09-15 07:23:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:23:31.005332949 +0000 UTC m=+315.551903064" watchObservedRunningTime="2026-09-15 07:23:31.006810962 +0000 UTC m=+315.553381108"
Sep 15 07:23:31.092273 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:31.092216    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:23:39.260535 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:39.260497    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:39.260968 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:39.260567    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:42.270583 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.270530    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4" podUID="b8697c4f-de34-4c3f-ae67-fbbb429010c0" containerName="registry" containerID="cri-o://c85b12c569fac539e1224c1de2b9a6285ff2159de01c30991d384b380ff046d0" gracePeriod=30
Sep 15 07:23:42.515182 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.515158    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:23:42.611107 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.611023    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b8697c4f-de34-4c3f-ae67-fbbb429010c0-image-registry-private-configuration\") pod \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") "
Sep 15 07:23:42.611107 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.611068    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b8697c4f-de34-4c3f-ae67-fbbb429010c0-installation-pull-secrets\") pod \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") "
Sep 15 07:23:42.611107 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.611096    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b8697c4f-de34-4c3f-ae67-fbbb429010c0-ca-trust-extracted\") pod \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") "
Sep 15 07:23:42.611394 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.611142    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmrrh\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-kube-api-access-wmrrh\") pod \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") "
Sep 15 07:23:42.611394 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.611160    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b8697c4f-de34-4c3f-ae67-fbbb429010c0-trusted-ca\") pod \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") "
Sep 15 07:23:42.611394 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.611194    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-bound-sa-token\") pod \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") "
Sep 15 07:23:42.611394 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.611227    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls\") pod \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") "
Sep 15 07:23:42.611394 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.611294    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-certificates\") pod \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\" (UID: \"b8697c4f-de34-4c3f-ae67-fbbb429010c0\") "
Sep 15 07:23:42.611917 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.611836    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8697c4f-de34-4c3f-ae67-fbbb429010c0-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b8697c4f-de34-4c3f-ae67-fbbb429010c0" (UID: "b8697c4f-de34-4c3f-ae67-fbbb429010c0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:23:42.611917 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.611879    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b8697c4f-de34-4c3f-ae67-fbbb429010c0" (UID: "b8697c4f-de34-4c3f-ae67-fbbb429010c0"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:23:42.615655 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.615588    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8697c4f-de34-4c3f-ae67-fbbb429010c0-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b8697c4f-de34-4c3f-ae67-fbbb429010c0" (UID: "b8697c4f-de34-4c3f-ae67-fbbb429010c0"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:23:42.615655 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.615638    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b8697c4f-de34-4c3f-ae67-fbbb429010c0" (UID: "b8697c4f-de34-4c3f-ae67-fbbb429010c0"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:23:42.615811 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.615694    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8697c4f-de34-4c3f-ae67-fbbb429010c0-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b8697c4f-de34-4c3f-ae67-fbbb429010c0" (UID: "b8697c4f-de34-4c3f-ae67-fbbb429010c0"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:23:42.617427 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.617395    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-kube-api-access-wmrrh" (OuterVolumeSpecName: "kube-api-access-wmrrh") pod "b8697c4f-de34-4c3f-ae67-fbbb429010c0" (UID: "b8697c4f-de34-4c3f-ae67-fbbb429010c0"). InnerVolumeSpecName "kube-api-access-wmrrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:23:42.620554 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.620442    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b8697c4f-de34-4c3f-ae67-fbbb429010c0" (UID: "b8697c4f-de34-4c3f-ae67-fbbb429010c0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:23:42.622180 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.622153    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8697c4f-de34-4c3f-ae67-fbbb429010c0-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b8697c4f-de34-4c3f-ae67-fbbb429010c0" (UID: "b8697c4f-de34-4c3f-ae67-fbbb429010c0"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:23:42.712572 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.712535    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-tls\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:23:42.712572 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.712566    2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b8697c4f-de34-4c3f-ae67-fbbb429010c0-registry-certificates\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:23:42.712765 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.712581    2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b8697c4f-de34-4c3f-ae67-fbbb429010c0-image-registry-private-configuration\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:23:42.712765 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.712595    2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b8697c4f-de34-4c3f-ae67-fbbb429010c0-installation-pull-secrets\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:23:42.712765 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.712607    2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b8697c4f-de34-4c3f-ae67-fbbb429010c0-ca-trust-extracted\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:23:42.712765 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.712619    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmrrh\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-kube-api-access-wmrrh\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:23:42.712765 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.712629    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b8697c4f-de34-4c3f-ae67-fbbb429010c0-trusted-ca\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:23:42.712765 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:42.712640    2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b8697c4f-de34-4c3f-ae67-fbbb429010c0-bound-sa-token\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:23:43.025660 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:43.025617    2571 generic.go:356] "Generic (PLEG): container finished" podID="b8697c4f-de34-4c3f-ae67-fbbb429010c0" containerID="c85b12c569fac539e1224c1de2b9a6285ff2159de01c30991d384b380ff046d0" exitCode=0
Sep 15 07:23:43.025829 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:43.025674    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4"
Sep 15 07:23:43.025829 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:43.025677    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4" event={"ID":"b8697c4f-de34-4c3f-ae67-fbbb429010c0","Type":"ContainerDied","Data":"c85b12c569fac539e1224c1de2b9a6285ff2159de01c30991d384b380ff046d0"}
Sep 15 07:23:43.025829 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:43.025792    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bd85df4cd-n9nj4" event={"ID":"b8697c4f-de34-4c3f-ae67-fbbb429010c0","Type":"ContainerDied","Data":"ebfe2023e520ffd254fb27dc0473b5ceca6db0626f67c056d59951d9d5246799"}
Sep 15 07:23:43.025829 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:43.025813    2571 scope.go:117] "RemoveContainer" containerID="c85b12c569fac539e1224c1de2b9a6285ff2159de01c30991d384b380ff046d0"
Sep 15 07:23:43.036757 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:43.036735    2571 scope.go:117] "RemoveContainer" containerID="c85b12c569fac539e1224c1de2b9a6285ff2159de01c30991d384b380ff046d0"
Sep 15 07:23:43.037035 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:23:43.037017    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c85b12c569fac539e1224c1de2b9a6285ff2159de01c30991d384b380ff046d0\": container with ID starting with c85b12c569fac539e1224c1de2b9a6285ff2159de01c30991d384b380ff046d0 not found: ID does not exist" containerID="c85b12c569fac539e1224c1de2b9a6285ff2159de01c30991d384b380ff046d0"
Sep 15 07:23:43.037090 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:43.037043    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c85b12c569fac539e1224c1de2b9a6285ff2159de01c30991d384b380ff046d0"} err="failed to get container status \"c85b12c569fac539e1224c1de2b9a6285ff2159de01c30991d384b380ff046d0\": rpc error: code = NotFound desc = could not find container \"c85b12c569fac539e1224c1de2b9a6285ff2159de01c30991d384b380ff046d0\": container with ID starting with c85b12c569fac539e1224c1de2b9a6285ff2159de01c30991d384b380ff046d0 not found: ID does not exist"
Sep 15 07:23:43.058096 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:43.058071    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bd85df4cd-n9nj4"]
Sep 15 07:23:43.067404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:43.067382    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5bd85df4cd-n9nj4"]
Sep 15 07:23:43.908375 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:43.908343    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8697c4f-de34-4c3f-ae67-fbbb429010c0" path="/var/lib/kubelet/pods/b8697c4f-de34-4c3f-ae67-fbbb429010c0/volumes"
Sep 15 07:23:53.064888 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:53.064846    2571 generic.go:356] "Generic (PLEG): container finished" podID="fb9280f4-141e-41f1-8e67-4d08778d723f" containerID="3e36267b3230249c293c7d49708b2132e14825bf2baa16b19d530bb66203430f" exitCode=0
Sep 15 07:23:53.065345 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:53.064911    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kc5ms" event={"ID":"fb9280f4-141e-41f1-8e67-4d08778d723f","Type":"ContainerDied","Data":"3e36267b3230249c293c7d49708b2132e14825bf2baa16b19d530bb66203430f"}
Sep 15 07:23:53.065345 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:53.065287    2571 scope.go:117] "RemoveContainer" containerID="3e36267b3230249c293c7d49708b2132e14825bf2baa16b19d530bb66203430f"
Sep 15 07:23:54.072294 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:54.072224    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kc5ms" event={"ID":"fb9280f4-141e-41f1-8e67-4d08778d723f","Type":"ContainerStarted","Data":"8cc2fd53f2b8573c324f7c6cf43b06433cee5594bbf75e8ffa2761265621839a"}
Sep 15 07:23:56.085272 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:56.085222    2571 generic.go:356] "Generic (PLEG): container finished" podID="ea4c7c35-ce08-453e-b629-f52c0ab26c66" containerID="2169f093e51a106770f358c6a1eba3f66cd3daf0b88fd3922aa8931bb8ba20f7" exitCode=0
Sep 15 07:23:56.085934 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:56.085909    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-j7dlf" event={"ID":"ea4c7c35-ce08-453e-b629-f52c0ab26c66","Type":"ContainerDied","Data":"2169f093e51a106770f358c6a1eba3f66cd3daf0b88fd3922aa8931bb8ba20f7"}
Sep 15 07:23:56.086548 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:56.086530    2571 scope.go:117] "RemoveContainer" containerID="2169f093e51a106770f358c6a1eba3f66cd3daf0b88fd3922aa8931bb8ba20f7"
Sep 15 07:23:57.091867 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:57.091834    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-j7dlf" event={"ID":"ea4c7c35-ce08-453e-b629-f52c0ab26c66","Type":"ContainerStarted","Data":"00e868c426ec8c3c5be52d186c71c3b1533c6bd40e86397b1d1ff3a3533c87ef"}
Sep 15 07:23:59.265588 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:59.265557    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:23:59.269352 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:23:59.269331    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-8b6949c-j2xf4"
Sep 15 07:24:21.092593 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:21.092544    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:21.111457 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:21.111431    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:21.204329 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:21.204300    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:29.833596 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:29.833555    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs\") pod \"network-metrics-daemon-jhh6r\" (UID: \"bf0b2425-be3b-4095-b8ab-a44f426176ac\") " pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:24:29.835939 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:29.835913    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf0b2425-be3b-4095-b8ab-a44f426176ac-metrics-certs\") pod \"network-metrics-daemon-jhh6r\" (UID: \"bf0b2425-be3b-4095-b8ab-a44f426176ac\") " pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:24:29.905901 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:29.905868    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ftj64\""
Sep 15 07:24:29.913649 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:29.913629    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhh6r"
Sep 15 07:24:30.181609 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:30.181586    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jhh6r"]
Sep 15 07:24:30.184380 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:24:30.184351    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf0b2425_be3b_4095_b8ab_a44f426176ac.slice/crio-a9cafbd509fcbf65aefa3d975783812eced131993f8c9eb6b8a98ac41cd19546 WatchSource:0}: Error finding container a9cafbd509fcbf65aefa3d975783812eced131993f8c9eb6b8a98ac41cd19546: Status 404 returned error can't find the container with id a9cafbd509fcbf65aefa3d975783812eced131993f8c9eb6b8a98ac41cd19546
Sep 15 07:24:30.223846 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:30.223814    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jhh6r" event={"ID":"bf0b2425-be3b-4095-b8ab-a44f426176ac","Type":"ContainerStarted","Data":"a9cafbd509fcbf65aefa3d975783812eced131993f8c9eb6b8a98ac41cd19546"}
Sep 15 07:24:32.235207 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:32.235168    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jhh6r" event={"ID":"bf0b2425-be3b-4095-b8ab-a44f426176ac","Type":"ContainerStarted","Data":"2b4adadffe310917a692845e29713f14de4c03f9a101b52460eb310263381b29"}
Sep 15 07:24:32.235621 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:32.235225    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jhh6r" event={"ID":"bf0b2425-be3b-4095-b8ab-a44f426176ac","Type":"ContainerStarted","Data":"65befe0c0e7192d6e763595802545d2ab1eaa82921085e03927a89ff9f5576fd"}
Sep 15 07:24:32.250658 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:32.250608    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jhh6r" podStartSLOduration=375.049905034 podStartE2EDuration="6m16.250593505s" totalImagesPullingTime="1.200688471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:24:32.249190795 +0000 UTC m=+376.795760917" watchObservedRunningTime="2026-09-15 07:24:32.250593505 +0000 UTC m=+376.797163654"
Sep 15 07:24:39.177054 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.177015    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 07:24:39.177517 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.177300    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="prometheus" containerID="cri-o://4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f" gracePeriod=600
Sep 15 07:24:39.177517 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.177348    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="config-reloader" containerID="cri-o://eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff" gracePeriod=600
Sep 15 07:24:39.177517 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.177348    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="thanos-sidecar" containerID="cri-o://0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb" gracePeriod=600
Sep 15 07:24:39.177517 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.177395    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="kube-rbac-proxy-web" containerID="cri-o://6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78" gracePeriod=600
Sep 15 07:24:39.177517 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.177400    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="kube-rbac-proxy-thanos" containerID="cri-o://7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e" gracePeriod=600
Sep 15 07:24:39.177517 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.177323    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="kube-rbac-proxy" containerID="cri-o://917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880" gracePeriod=600
Sep 15 07:24:39.415266 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.415233    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:39.508337 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.508301    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-web-config\") pod \"d996759f-03a2-4cba-804e-ab4ae2468df1\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") "
Sep 15 07:24:39.508495 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.508355    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-configmap-metrics-client-ca\") pod \"d996759f-03a2-4cba-804e-ab4ae2468df1\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") "
Sep 15 07:24:39.508495 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.508394    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-rulefiles-0\") pod \"d996759f-03a2-4cba-804e-ab4ae2468df1\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") "
Sep 15 07:24:39.508495 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.508431    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-configmap-kubelet-serving-ca-bundle\") pod \"d996759f-03a2-4cba-804e-ab4ae2468df1\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") "
Sep 15 07:24:39.508495 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.508458    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-trusted-ca-bundle\") pod \"d996759f-03a2-4cba-804e-ab4ae2468df1\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") "
Sep 15 07:24:39.508682 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.508496    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-prometheus-k8s-tls\") pod \"d996759f-03a2-4cba-804e-ab4ae2468df1\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") "
Sep 15 07:24:39.508682 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.508525    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-rulefiles-2\") pod \"d996759f-03a2-4cba-804e-ab4ae2468df1\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") "
Sep 15 07:24:39.508682 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.508553    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"d996759f-03a2-4cba-804e-ab4ae2468df1\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") "
Sep 15 07:24:39.508682 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.508576    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjpb9\" (UniqueName: \"kubernetes.io/projected/d996759f-03a2-4cba-804e-ab4ae2468df1-kube-api-access-vjpb9\") pod \"d996759f-03a2-4cba-804e-ab4ae2468df1\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") "
Sep 15 07:24:39.508682 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.508621    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d996759f-03a2-4cba-804e-ab4ae2468df1-config-out\") pod \"d996759f-03a2-4cba-804e-ab4ae2468df1\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") "
Sep 15 07:24:39.508682 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.508647    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-configmap-serving-certs-ca-bundle\") pod \"d996759f-03a2-4cba-804e-ab4ae2468df1\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") "
Sep 15 07:24:39.508682 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.508675    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-config\") pod \"d996759f-03a2-4cba-804e-ab4ae2468df1\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") "
Sep 15 07:24:39.509089 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.508700    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"d996759f-03a2-4cba-804e-ab4ae2468df1\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") "
Sep 15 07:24:39.509089 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.508725    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-db\") pod \"d996759f-03a2-4cba-804e-ab4ae2468df1\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") "
Sep 15 07:24:39.509089 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.508755    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d996759f-03a2-4cba-804e-ab4ae2468df1-tls-assets\") pod \"d996759f-03a2-4cba-804e-ab4ae2468df1\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") "
Sep 15 07:24:39.509089 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.508794    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-metrics-client-certs\") pod \"d996759f-03a2-4cba-804e-ab4ae2468df1\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") "
Sep 15 07:24:39.509089 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.508817    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-kube-rbac-proxy\") pod \"d996759f-03a2-4cba-804e-ab4ae2468df1\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") "
Sep 15 07:24:39.509089 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.508836    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-thanos-prometheus-http-client-file\") pod \"d996759f-03a2-4cba-804e-ab4ae2468df1\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") "
Sep 15 07:24:39.509089 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.508867    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-rulefiles-1\") pod \"d996759f-03a2-4cba-804e-ab4ae2468df1\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") "
Sep 15 07:24:39.509089 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.508890    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-grpc-tls\") pod \"d996759f-03a2-4cba-804e-ab4ae2468df1\" (UID: \"d996759f-03a2-4cba-804e-ab4ae2468df1\") "
Sep 15 07:24:39.509089 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.508890    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "d996759f-03a2-4cba-804e-ab4ae2468df1" (UID: "d996759f-03a2-4cba-804e-ab4ae2468df1"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:24:39.509546 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.509124    2571 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:24:39.509546 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.509373    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "d996759f-03a2-4cba-804e-ab4ae2468df1" (UID: "d996759f-03a2-4cba-804e-ab4ae2468df1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:24:39.509546 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.509529    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "d996759f-03a2-4cba-804e-ab4ae2468df1" (UID: "d996759f-03a2-4cba-804e-ab4ae2468df1"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:24:39.510775 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.509828    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "d996759f-03a2-4cba-804e-ab4ae2468df1" (UID: "d996759f-03a2-4cba-804e-ab4ae2468df1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:24:39.510775 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.510734    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "d996759f-03a2-4cba-804e-ab4ae2468df1" (UID: "d996759f-03a2-4cba-804e-ab4ae2468df1"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:24:39.511908 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.511792    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "d996759f-03a2-4cba-804e-ab4ae2468df1" (UID: "d996759f-03a2-4cba-804e-ab4ae2468df1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:24:39.512049 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.512000    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "d996759f-03a2-4cba-804e-ab4ae2468df1" (UID: "d996759f-03a2-4cba-804e-ab4ae2468df1"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:24:39.512685 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.512647    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "d996759f-03a2-4cba-804e-ab4ae2468df1" (UID: "d996759f-03a2-4cba-804e-ab4ae2468df1"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:24:39.512775 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.512699    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "d996759f-03a2-4cba-804e-ab4ae2468df1" (UID: "d996759f-03a2-4cba-804e-ab4ae2468df1"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:24:39.512948 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.512907    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d996759f-03a2-4cba-804e-ab4ae2468df1-config-out" (OuterVolumeSpecName: "config-out") pod "d996759f-03a2-4cba-804e-ab4ae2468df1" (UID: "d996759f-03a2-4cba-804e-ab4ae2468df1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:24:39.512948 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.512912    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "d996759f-03a2-4cba-804e-ab4ae2468df1" (UID: "d996759f-03a2-4cba-804e-ab4ae2468df1"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:24:39.513075 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.512950    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d996759f-03a2-4cba-804e-ab4ae2468df1-kube-api-access-vjpb9" (OuterVolumeSpecName: "kube-api-access-vjpb9") pod "d996759f-03a2-4cba-804e-ab4ae2468df1" (UID: "d996759f-03a2-4cba-804e-ab4ae2468df1"). InnerVolumeSpecName "kube-api-access-vjpb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:24:39.513075 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.513054    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "d996759f-03a2-4cba-804e-ab4ae2468df1" (UID: "d996759f-03a2-4cba-804e-ab4ae2468df1"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:24:39.513269 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.513233    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "d996759f-03a2-4cba-804e-ab4ae2468df1" (UID: "d996759f-03a2-4cba-804e-ab4ae2468df1"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:24:39.513766 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.513745    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "d996759f-03a2-4cba-804e-ab4ae2468df1" (UID: "d996759f-03a2-4cba-804e-ab4ae2468df1"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:24:39.513855 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.513833    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "d996759f-03a2-4cba-804e-ab4ae2468df1" (UID: "d996759f-03a2-4cba-804e-ab4ae2468df1"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:24:39.513954 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.513936    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d996759f-03a2-4cba-804e-ab4ae2468df1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d996759f-03a2-4cba-804e-ab4ae2468df1" (UID: "d996759f-03a2-4cba-804e-ab4ae2468df1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:24:39.514531 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.514516    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-config" (OuterVolumeSpecName: "config") pod "d996759f-03a2-4cba-804e-ab4ae2468df1" (UID: "d996759f-03a2-4cba-804e-ab4ae2468df1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:24:39.514993 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.514976    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "d996759f-03a2-4cba-804e-ab4ae2468df1" (UID: "d996759f-03a2-4cba-804e-ab4ae2468df1"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:24:39.522709 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.522691    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-web-config" (OuterVolumeSpecName: "web-config") pod "d996759f-03a2-4cba-804e-ab4ae2468df1" (UID: "d996759f-03a2-4cba-804e-ab4ae2468df1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:24:39.610288 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.610233    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:24:39.610288 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.610284    2571 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-grpc-tls\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:24:39.610288 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.610294    2571 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-web-config\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:24:39.610502 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.610302    2571 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-configmap-metrics-client-ca\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:24:39.610502 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.610311    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:24:39.610502 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.610321    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:24:39.610502 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.610329    2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:24:39.610502 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.610338    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:24:39.610502 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.610347    2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:24:39.610502 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.610355    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjpb9\" (UniqueName: \"kubernetes.io/projected/d996759f-03a2-4cba-804e-ab4ae2468df1-kube-api-access-vjpb9\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:24:39.610502 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.610363    2571 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d996759f-03a2-4cba-804e-ab4ae2468df1-config-out\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:24:39.610502 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.610372    2571 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d996759f-03a2-4cba-804e-ab4ae2468df1-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:24:39.610502 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.610381    2571 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-config\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:24:39.610502 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.610389    2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:24:39.610502 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.610399    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d996759f-03a2-4cba-804e-ab4ae2468df1-prometheus-k8s-db\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:24:39.610502 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.610408    2571 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d996759f-03a2-4cba-804e-ab4ae2468df1-tls-assets\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:24:39.610502 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.610416    2571 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-metrics-client-certs\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:24:39.610502 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.610424    2571 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-secret-kube-rbac-proxy\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:24:39.610502 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:39.610432    2571 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d996759f-03a2-4cba-804e-ab4ae2468df1-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:24:40.270032 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.269998    2571 generic.go:356] "Generic (PLEG): container finished" podID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerID="7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e" exitCode=0
Sep 15 07:24:40.270032 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.270026    2571 generic.go:356] "Generic (PLEG): container finished" podID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerID="917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880" exitCode=0
Sep 15 07:24:40.270032 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.270032    2571 generic.go:356] "Generic (PLEG): container finished" podID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerID="6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78" exitCode=0
Sep 15 07:24:40.270032 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.270038    2571 generic.go:356] "Generic (PLEG): container finished" podID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerID="0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb" exitCode=0
Sep 15 07:24:40.270576 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.270045    2571 generic.go:356] "Generic (PLEG): container finished" podID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerID="eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff" exitCode=0
Sep 15 07:24:40.270576 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.270050    2571 generic.go:356] "Generic (PLEG): container finished" podID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerID="4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f" exitCode=0
Sep 15 07:24:40.270576 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.270071    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.270576 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.270079    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d996759f-03a2-4cba-804e-ab4ae2468df1","Type":"ContainerDied","Data":"7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e"}
Sep 15 07:24:40.270576 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.270156    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d996759f-03a2-4cba-804e-ab4ae2468df1","Type":"ContainerDied","Data":"917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880"}
Sep 15 07:24:40.270576 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.270187    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d996759f-03a2-4cba-804e-ab4ae2468df1","Type":"ContainerDied","Data":"6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78"}
Sep 15 07:24:40.270576 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.270218    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d996759f-03a2-4cba-804e-ab4ae2468df1","Type":"ContainerDied","Data":"0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb"}
Sep 15 07:24:40.270576 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.270267    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d996759f-03a2-4cba-804e-ab4ae2468df1","Type":"ContainerDied","Data":"eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff"}
Sep 15 07:24:40.270576 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.270302    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d996759f-03a2-4cba-804e-ab4ae2468df1","Type":"ContainerDied","Data":"4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f"}
Sep 15 07:24:40.270576 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.270327    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d996759f-03a2-4cba-804e-ab4ae2468df1","Type":"ContainerDied","Data":"7250793c9a4d974a197e29866d6fe5594a7b99b9103d22347fa801be07de67c0"}
Sep 15 07:24:40.270576 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.270221    2571 scope.go:117] "RemoveContainer" containerID="7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e"
Sep 15 07:24:40.278590 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.278576    2571 scope.go:117] "RemoveContainer" containerID="917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880"
Sep 15 07:24:40.285685 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.285670    2571 scope.go:117] "RemoveContainer" containerID="6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78"
Sep 15 07:24:40.296002 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.295971    2571 scope.go:117] "RemoveContainer" containerID="0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb"
Sep 15 07:24:40.302830 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.302808    2571 scope.go:117] "RemoveContainer" containerID="eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff"
Sep 15 07:24:40.306661 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.306641    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 07:24:40.310049 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.310030    2571 scope.go:117] "RemoveContainer" containerID="4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f"
Sep 15 07:24:40.314067 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.314043    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 07:24:40.317749 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.317728    2571 scope.go:117] "RemoveContainer" containerID="58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61"
Sep 15 07:24:40.324304 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.324284    2571 scope.go:117] "RemoveContainer" containerID="7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e"
Sep 15 07:24:40.324584 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:24:40.324563    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e\": container with ID starting with 7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e not found: ID does not exist" containerID="7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e"
Sep 15 07:24:40.324636 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.324592    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e"} err="failed to get container status \"7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e\": rpc error: code = NotFound desc = could not find container \"7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e\": container with ID starting with 7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e not found: ID does not exist"
Sep 15 07:24:40.324636 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.324610    2571 scope.go:117] "RemoveContainer" containerID="917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880"
Sep 15 07:24:40.324841 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:24:40.324824    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880\": container with ID starting with 917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880 not found: ID does not exist" containerID="917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880"
Sep 15 07:24:40.324874 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.324847    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880"} err="failed to get container status \"917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880\": rpc error: code = NotFound desc = could not find container \"917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880\": container with ID starting with 917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880 not found: ID does not exist"
Sep 15 07:24:40.324874 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.324861    2571 scope.go:117] "RemoveContainer" containerID="6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78"
Sep 15 07:24:40.325092 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:24:40.325079    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78\": container with ID starting with 6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78 not found: ID does not exist" containerID="6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78"
Sep 15 07:24:40.325127 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.325094    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78"} err="failed to get container status \"6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78\": rpc error: code = NotFound desc = could not find container \"6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78\": container with ID starting with 6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78 not found: ID does not exist"
Sep 15 07:24:40.325127 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.325104    2571 scope.go:117] "RemoveContainer" containerID="0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb"
Sep 15 07:24:40.325321 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:24:40.325305    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb\": container with ID starting with 0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb not found: ID does not exist" containerID="0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb"
Sep 15 07:24:40.325362 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.325328    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb"} err="failed to get container status \"0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb\": rpc error: code = NotFound desc = could not find container \"0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb\": container with ID starting with 0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb not found: ID does not exist"
Sep 15 07:24:40.325362 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.325344    2571 scope.go:117] "RemoveContainer" containerID="eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff"
Sep 15 07:24:40.325576 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:24:40.325562    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff\": container with ID starting with eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff not found: ID does not exist" containerID="eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff"
Sep 15 07:24:40.325610 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.325579    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff"} err="failed to get container status \"eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff\": rpc error: code = NotFound desc = could not find container \"eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff\": container with ID starting with eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff not found: ID does not exist"
Sep 15 07:24:40.325610 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.325599    2571 scope.go:117] "RemoveContainer" containerID="4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f"
Sep 15 07:24:40.325832 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:24:40.325811    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f\": container with ID starting with 4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f not found: ID does not exist" containerID="4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f"
Sep 15 07:24:40.325920 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.325834    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f"} err="failed to get container status \"4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f\": rpc error: code = NotFound desc = could not find container \"4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f\": container with ID starting with 4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f not found: ID does not exist"
Sep 15 07:24:40.325920 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.325849    2571 scope.go:117] "RemoveContainer" containerID="58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61"
Sep 15 07:24:40.326069 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:24:40.326052    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61\": container with ID starting with 58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61 not found: ID does not exist" containerID="58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61"
Sep 15 07:24:40.326126 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.326075    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61"} err="failed to get container status \"58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61\": rpc error: code = NotFound desc = could not find container \"58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61\": container with ID starting with 58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61 not found: ID does not exist"
Sep 15 07:24:40.326126 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.326092    2571 scope.go:117] "RemoveContainer" containerID="7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e"
Sep 15 07:24:40.326315 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.326300    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e"} err="failed to get container status \"7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e\": rpc error: code = NotFound desc = could not find container \"7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e\": container with ID starting with 7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e not found: ID does not exist"
Sep 15 07:24:40.326315 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.326315    2571 scope.go:117] "RemoveContainer" containerID="917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880"
Sep 15 07:24:40.326534 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.326518    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880"} err="failed to get container status \"917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880\": rpc error: code = NotFound desc = could not find container \"917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880\": container with ID starting with 917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880 not found: ID does not exist"
Sep 15 07:24:40.326534 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.326533    2571 scope.go:117] "RemoveContainer" containerID="6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78"
Sep 15 07:24:40.326706 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.326693    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78"} err="failed to get container status \"6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78\": rpc error: code = NotFound desc = could not find container \"6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78\": container with ID starting with 6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78 not found: ID does not exist"
Sep 15 07:24:40.326706 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.326706    2571 scope.go:117] "RemoveContainer" containerID="0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb"
Sep 15 07:24:40.326875 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.326861    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb"} err="failed to get container status \"0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb\": rpc error: code = NotFound desc = could not find container \"0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb\": container with ID starting with 0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb not found: ID does not exist"
Sep 15 07:24:40.326919 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.326875    2571 scope.go:117] "RemoveContainer" containerID="eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff"
Sep 15 07:24:40.327085 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.327070    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff"} err="failed to get container status \"eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff\": rpc error: code = NotFound desc = could not find container \"eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff\": container with ID starting with eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff not found: ID does not exist"
Sep 15 07:24:40.327085 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.327085    2571 scope.go:117] "RemoveContainer" containerID="4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f"
Sep 15 07:24:40.327305 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.327285    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f"} err="failed to get container status \"4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f\": rpc error: code = NotFound desc = could not find container \"4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f\": container with ID starting with 4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f not found: ID does not exist"
Sep 15 07:24:40.327343 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.327306    2571 scope.go:117] "RemoveContainer" containerID="58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61"
Sep 15 07:24:40.327513 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.327497    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61"} err="failed to get container status \"58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61\": rpc error: code = NotFound desc = could not find container \"58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61\": container with ID starting with 58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61 not found: ID does not exist"
Sep 15 07:24:40.327555 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.327513    2571 scope.go:117] "RemoveContainer" containerID="7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e"
Sep 15 07:24:40.327696 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.327681    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e"} err="failed to get container status \"7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e\": rpc error: code = NotFound desc = could not find container \"7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e\": container with ID starting with 7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e not found: ID does not exist"
Sep 15 07:24:40.327734 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.327696    2571 scope.go:117] "RemoveContainer" containerID="917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880"
Sep 15 07:24:40.327883 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.327867    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880"} err="failed to get container status \"917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880\": rpc error: code = NotFound desc = could not find container \"917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880\": container with ID starting with 917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880 not found: ID does not exist"
Sep 15 07:24:40.327921 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.327882    2571 scope.go:117] "RemoveContainer" containerID="6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78"
Sep 15 07:24:40.328061 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.328046    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78"} err="failed to get container status \"6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78\": rpc error: code = NotFound desc = could not find container \"6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78\": container with ID starting with 6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78 not found: ID does not exist"
Sep 15 07:24:40.328098 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.328061    2571 scope.go:117] "RemoveContainer" containerID="0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb"
Sep 15 07:24:40.328233 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.328219    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb"} err="failed to get container status \"0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb\": rpc error: code = NotFound desc = could not find container \"0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb\": container with ID starting with 0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb not found: ID does not exist"
Sep 15 07:24:40.328287 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.328233    2571 scope.go:117] "RemoveContainer" containerID="eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff"
Sep 15 07:24:40.328427 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.328410    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff"} err="failed to get container status \"eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff\": rpc error: code = NotFound desc = could not find container \"eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff\": container with ID starting with eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff not found: ID does not exist"
Sep 15 07:24:40.328465 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.328429    2571 scope.go:117] "RemoveContainer" containerID="4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f"
Sep 15 07:24:40.328660 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.328644    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f"} err="failed to get container status \"4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f\": rpc error: code = NotFound desc = could not find container \"4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f\": container with ID starting with 4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f not found: ID does not exist"
Sep 15 07:24:40.328660 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.328659    2571 scope.go:117] "RemoveContainer" containerID="58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61"
Sep 15 07:24:40.328886 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.328866    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61"} err="failed to get container status \"58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61\": rpc error: code = NotFound desc = could not find container \"58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61\": container with ID starting with 58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61 not found: ID does not exist"
Sep 15 07:24:40.328924 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.328887    2571 scope.go:117] "RemoveContainer" containerID="7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e"
Sep 15 07:24:40.329102 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.329087    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e"} err="failed to get container status \"7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e\": rpc error: code = NotFound desc = could not find container \"7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e\": container with ID starting with 7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e not found: ID does not exist"
Sep 15 07:24:40.329139 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.329103    2571 scope.go:117] "RemoveContainer" containerID="917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880"
Sep 15 07:24:40.329311 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.329295    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880"} err="failed to get container status \"917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880\": rpc error: code = NotFound desc = could not find container \"917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880\": container with ID starting with 917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880 not found: ID does not exist"
Sep 15 07:24:40.329358 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.329311    2571 scope.go:117] "RemoveContainer" containerID="6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78"
Sep 15 07:24:40.329491 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.329476    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78"} err="failed to get container status \"6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78\": rpc error: code = NotFound desc = could not find container \"6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78\": container with ID starting with 6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78 not found: ID does not exist"
Sep 15 07:24:40.329531 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.329491    2571 scope.go:117] "RemoveContainer" containerID="0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb"
Sep 15 07:24:40.329669 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.329653    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb"} err="failed to get container status \"0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb\": rpc error: code = NotFound desc = could not find container \"0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb\": container with ID starting with 0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb not found: ID does not exist"
Sep 15 07:24:40.329724 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.329668    2571 scope.go:117] "RemoveContainer" containerID="eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff"
Sep 15 07:24:40.329902 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.329885    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff"} err="failed to get container status \"eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff\": rpc error: code = NotFound desc = could not find container \"eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff\": container with ID starting with eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff not found: ID does not exist"
Sep 15 07:24:40.329953 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.329903    2571 scope.go:117] "RemoveContainer" containerID="4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f"
Sep 15 07:24:40.330130 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.330115    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f"} err="failed to get container status \"4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f\": rpc error: code = NotFound desc = could not find container \"4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f\": container with ID starting with 4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f not found: ID does not exist"
Sep 15 07:24:40.330130 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.330130    2571 scope.go:117] "RemoveContainer" containerID="58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61"
Sep 15 07:24:40.330339 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.330324    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61"} err="failed to get container status \"58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61\": rpc error: code = NotFound desc = could not find container \"58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61\": container with ID starting with 58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61 not found: ID does not exist"
Sep 15 07:24:40.330386 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.330340    2571 scope.go:117] "RemoveContainer" containerID="7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e"
Sep 15 07:24:40.330515 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.330498    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e"} err="failed to get container status \"7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e\": rpc error: code = NotFound desc = could not find container \"7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e\": container with ID starting with 7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e not found: ID does not exist"
Sep 15 07:24:40.330565 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.330515    2571 scope.go:117] "RemoveContainer" containerID="917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880"
Sep 15 07:24:40.330733 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.330717    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880"} err="failed to get container status \"917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880\": rpc error: code = NotFound desc = could not find container \"917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880\": container with ID starting with 917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880 not found: ID does not exist"
Sep 15 07:24:40.330733 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.330732    2571 scope.go:117] "RemoveContainer" containerID="6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78"
Sep 15 07:24:40.330918 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.330904    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78"} err="failed to get container status \"6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78\": rpc error: code = NotFound desc = could not find container \"6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78\": container with ID starting with 6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78 not found: ID does not exist"
Sep 15 07:24:40.330956 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.330918    2571 scope.go:117] "RemoveContainer" containerID="0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb"
Sep 15 07:24:40.331128 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.331110    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb"} err="failed to get container status \"0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb\": rpc error: code = NotFound desc = could not find container \"0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb\": container with ID starting with 0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb not found: ID does not exist"
Sep 15 07:24:40.331128 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.331127    2571 scope.go:117] "RemoveContainer" containerID="eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff"
Sep 15 07:24:40.331340 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.331322    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff"} err="failed to get container status \"eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff\": rpc error: code = NotFound desc = could not find container \"eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff\": container with ID starting with eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff not found: ID does not exist"
Sep 15 07:24:40.331378 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.331340    2571 scope.go:117] "RemoveContainer" containerID="4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f"
Sep 15 07:24:40.331546 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.331530    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f"} err="failed to get container status \"4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f\": rpc error: code = NotFound desc = could not find container \"4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f\": container with ID starting with 4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f not found: ID does not exist"
Sep 15 07:24:40.331546 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.331545    2571 scope.go:117] "RemoveContainer" containerID="58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61"
Sep 15 07:24:40.331738 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.331720    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61"} err="failed to get container status \"58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61\": rpc error: code = NotFound desc = could not find container \"58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61\": container with ID starting with 58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61 not found: ID does not exist"
Sep 15 07:24:40.331780 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.331737    2571 scope.go:117] "RemoveContainer" containerID="7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e"
Sep 15 07:24:40.331917 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.331902    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e"} err="failed to get container status \"7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e\": rpc error: code = NotFound desc = could not find container \"7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e\": container with ID starting with 7bc1cd137829220883bf25ba5ea44990aae871ba6011e8dfe051f070eece123e not found: ID does not exist"
Sep 15 07:24:40.332062 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.331918    2571 scope.go:117] "RemoveContainer" containerID="917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880"
Sep 15 07:24:40.332207 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.332139    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880"} err="failed to get container status \"917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880\": rpc error: code = NotFound desc = could not find container \"917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880\": container with ID starting with 917dcf4fa190a39608dce5fdbde83b536852551640b83ae623d2d55e8497c880 not found: ID does not exist"
Sep 15 07:24:40.332207 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.332158    2571 scope.go:117] "RemoveContainer" containerID="6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78"
Sep 15 07:24:40.332507 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.332461    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78"} err="failed to get container status \"6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78\": rpc error: code = NotFound desc = could not find container \"6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78\": container with ID starting with 6174f587c05b22560fd0a439f71a0dffa43f631bd1d0c219afe16242f5dccd78 not found: ID does not exist"
Sep 15 07:24:40.332507 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.332495    2571 scope.go:117] "RemoveContainer" containerID="0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb"
Sep 15 07:24:40.332807 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.332776    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb"} err="failed to get container status \"0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb\": rpc error: code = NotFound desc = could not find container \"0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb\": container with ID starting with 0e684d287cfdce43b993d48a72d65db7bc7111dfb716710a16844859730493cb not found: ID does not exist"
Sep 15 07:24:40.332863 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.332809    2571 scope.go:117] "RemoveContainer" containerID="eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff"
Sep 15 07:24:40.337649 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.333044    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff"} err="failed to get container status \"eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff\": rpc error: code = NotFound desc = could not find container \"eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff\": container with ID starting with eb79b0a63fbaa85d20c625f83d46f66002d3b6989f741a5cfd629c144600abff not found: ID does not exist"
Sep 15 07:24:40.337649 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.333070    2571 scope.go:117] "RemoveContainer" containerID="4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f"
Sep 15 07:24:40.337649 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.333327    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f"} err="failed to get container status \"4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f\": rpc error: code = NotFound desc = could not find container \"4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f\": container with ID starting with 4c2d50241b6159bc7dd287329992e2f53898906b58c0529b4feea2009ffcba0f not found: ID does not exist"
Sep 15 07:24:40.337649 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.333343    2571 scope.go:117] "RemoveContainer" containerID="58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61"
Sep 15 07:24:40.337649 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.333578    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61"} err="failed to get container status \"58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61\": rpc error: code = NotFound desc = could not find container \"58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61\": container with ID starting with 58db1090177810ab1278013c4c88a97071b7ddcacba58e4c0e68903d94f9da61 not found: ID does not exist"
Sep 15 07:24:40.344410 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344392    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 07:24:40.344736 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344723    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="kube-rbac-proxy-web"
Sep 15 07:24:40.344795 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344738    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="kube-rbac-proxy-web"
Sep 15 07:24:40.344795 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344750    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="prometheus"
Sep 15 07:24:40.344795 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344756    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="prometheus"
Sep 15 07:24:40.344795 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344761    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8697c4f-de34-4c3f-ae67-fbbb429010c0" containerName="registry"
Sep 15 07:24:40.344795 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344766    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8697c4f-de34-4c3f-ae67-fbbb429010c0" containerName="registry"
Sep 15 07:24:40.344795 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344777    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="thanos-sidecar"
Sep 15 07:24:40.344795 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344786    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="thanos-sidecar"
Sep 15 07:24:40.344795 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344793    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="kube-rbac-proxy"
Sep 15 07:24:40.345076 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344800    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="kube-rbac-proxy"
Sep 15 07:24:40.345076 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344807    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="config-reloader"
Sep 15 07:24:40.345076 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344812    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="config-reloader"
Sep 15 07:24:40.345076 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344824    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="init-config-reloader"
Sep 15 07:24:40.345076 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344828    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="init-config-reloader"
Sep 15 07:24:40.345076 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344834    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="kube-rbac-proxy-thanos"
Sep 15 07:24:40.345076 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344840    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="kube-rbac-proxy-thanos"
Sep 15 07:24:40.345076 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344897    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="kube-rbac-proxy-thanos"
Sep 15 07:24:40.345076 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344907    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="config-reloader"
Sep 15 07:24:40.345076 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344913    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="thanos-sidecar"
Sep 15 07:24:40.345076 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344918    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="prometheus"
Sep 15 07:24:40.345076 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344923    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="kube-rbac-proxy"
Sep 15 07:24:40.345076 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344930    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" containerName="kube-rbac-proxy-web"
Sep 15 07:24:40.345076 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.344938    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8697c4f-de34-4c3f-ae67-fbbb429010c0" containerName="registry"
Sep 15 07:24:40.350289 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.350274    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.353652 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.353634    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bkl0j842qou9r\""
Sep 15 07:24:40.354004 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.353987    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 07:24:40.354188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.354172    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fd9vs\""
Sep 15 07:24:40.354465 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.354448    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 07:24:40.354534 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.354479    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 07:24:40.354609 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.354554    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 07:24:40.354609 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.354568    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 07:24:40.356248 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.355981    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 07:24:40.356248 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.356042    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 07:24:40.356248 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.356067    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 07:24:40.356248 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.356231    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 07:24:40.356658 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.356644    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 07:24:40.356781 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.356649    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 07:24:40.358416 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.358387    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 07:24:40.358875 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.358857    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 07:24:40.361145 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.361129    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 07:24:40.372733 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.372709    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 07:24:40.416454 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.416422    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.416454 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.416454    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.416639 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.416473    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.416639 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.416543    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-config-out\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.416639 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.416574    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.416639 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.416597    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2cjh\" (UniqueName: \"kubernetes.io/projected/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-kube-api-access-w2cjh\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.416756 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.416639    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.416756 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.416669    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.416756 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.416683    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.416756 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.416703    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-web-config\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.416756 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.416723    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.416903 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.416764    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.416903 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.416784    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-config\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.416903 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.416800    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.416903 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.416819    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.416903 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.416833    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.416903 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.416874    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.416903 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.416900    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.417097 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.416928    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.417097 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.416953    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.517433 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.517386    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.517433 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.517433    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2cjh\" (UniqueName: \"kubernetes.io/projected/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-kube-api-access-w2cjh\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.517618 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.517457    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.517618 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.517483    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.517618 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.517498    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.517618 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.517517    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-web-config\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.517756 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.517709    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.517802 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.517760    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.517802 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.517783    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-config\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.517908 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.517806    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.517908 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.517837    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.517908 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.517860    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.517908 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.517903    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.518724 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.518695    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.518724 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.518690    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.518894 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.518769    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.518894 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.518813    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.518894 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.518844    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.518894 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.518875    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.519099 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.518908    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.519099 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.518936    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.519099 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.518979    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-config-out\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.519232 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.519120    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.519337 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.519315    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.521404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.519677    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.521404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.521050    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.521404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.521388    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-web-config\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.521849 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.521820    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.521949 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.521932    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.522019 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.521983    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.522077 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.522033    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-config\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.522077 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.522054    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.522353 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.522323    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.522506 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.522484    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.522930 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.522908    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-config-out\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.523031 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.522986    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.523649 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.523628    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.523862 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.523842    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.523903 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.523864    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.525884 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.525859    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2cjh\" (UniqueName: \"kubernetes.io/projected/e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b-kube-api-access-w2cjh\") pod \"prometheus-k8s-0\" (UID: \"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.660994 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.660959    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:24:40.943188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:40.943165    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 07:24:40.944524 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:24:40.944496    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4bf75cf_6cca_40b5_ae06_d3e6a9e3993b.slice/crio-663b7a53a627152316b7019d246ecfe47e215f1752ba07e23933cc148fd7be82 WatchSource:0}: Error finding container 663b7a53a627152316b7019d246ecfe47e215f1752ba07e23933cc148fd7be82: Status 404 returned error can't find the container with id 663b7a53a627152316b7019d246ecfe47e215f1752ba07e23933cc148fd7be82
Sep 15 07:24:41.275870 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:41.275834    2571 generic.go:356] "Generic (PLEG): container finished" podID="e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b" containerID="7fecbe498cb1483527ef620c84ce8ee098616e138b02ceb3f0a1b018991ce90b" exitCode=0
Sep 15 07:24:41.276331 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:41.275916    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b","Type":"ContainerDied","Data":"7fecbe498cb1483527ef620c84ce8ee098616e138b02ceb3f0a1b018991ce90b"}
Sep 15 07:24:41.276331 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:41.275982    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b","Type":"ContainerStarted","Data":"663b7a53a627152316b7019d246ecfe47e215f1752ba07e23933cc148fd7be82"}
Sep 15 07:24:41.912670 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:41.912590    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d996759f-03a2-4cba-804e-ab4ae2468df1" path="/var/lib/kubelet/pods/d996759f-03a2-4cba-804e-ab4ae2468df1/volumes"
Sep 15 07:24:42.284957 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:42.284920    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b","Type":"ContainerStarted","Data":"733b514bcff26d3c33d34e0150c3101de80674eee8ca0dcf05ea776d829c2db8"}
Sep 15 07:24:42.285371 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:42.284988    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b","Type":"ContainerStarted","Data":"534ff6ac16855e05f653e90c801cdd8e6ab29859d0a111b8783da9253733f30b"}
Sep 15 07:24:42.285371 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:42.285015    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b","Type":"ContainerStarted","Data":"8fc3c55feeee19ed426736fa22f6603333ccc355f35eb3294c209f3e49f0950e"}
Sep 15 07:24:42.285371 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:42.285053    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b","Type":"ContainerStarted","Data":"9eaeb1d75dacd21751a5d87952dff2f117f0f793fb4ab8681c776cdc468be625"}
Sep 15 07:24:42.285371 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:42.285081    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b","Type":"ContainerStarted","Data":"7784a3e0e2d1a2d6b354a3f7597527eca26408d0789e1f13cc7a1c876ac17250"}
Sep 15 07:24:42.285371 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:42.285106    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b","Type":"ContainerStarted","Data":"3416792c092492b72b2a36b8ea7ccac620f0f2d770e0a41525c967b2c23a2c0a"}
Sep 15 07:24:42.316403 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:42.316344    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.298646539 podStartE2EDuration="2.316322463s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.675924ms" isStatefulPod=false podCreationTimestamp="2026-09-15 07:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:24:42.313669597 +0000 UTC m=+386.860239708" watchObservedRunningTime="2026-09-15 07:24:42.316322463 +0000 UTC m=+386.862892563"
Sep 15 07:24:45.661320 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:24:45.661280    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:25:40.661651 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:25:40.661612    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:25:40.677114 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:25:40.677090    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:25:41.546739 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:25:41.546707    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 07:27:21.470121 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:21.470040    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dxfff"]
Sep 15 07:27:21.473556 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:21.473536    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dxfff"
Sep 15 07:27:21.475850 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:21.475830    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 15 07:27:21.475941 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:21.475864    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 15 07:27:21.476645 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:21.476626    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-hbtjz\""
Sep 15 07:27:21.483661 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:21.483643    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dxfff"]
Sep 15 07:27:21.579758 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:21.579725    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb107669-47a9-4a83-9268-3035b22408ce-bound-sa-token\") pod \"cert-manager-55b4d954d5-dxfff\" (UID: \"cb107669-47a9-4a83-9268-3035b22408ce\") " pod="cert-manager/cert-manager-55b4d954d5-dxfff"
Sep 15 07:27:21.579913 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:21.579792    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsxsz\" (UniqueName: \"kubernetes.io/projected/cb107669-47a9-4a83-9268-3035b22408ce-kube-api-access-hsxsz\") pod \"cert-manager-55b4d954d5-dxfff\" (UID: \"cb107669-47a9-4a83-9268-3035b22408ce\") " pod="cert-manager/cert-manager-55b4d954d5-dxfff"
Sep 15 07:27:21.680170 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:21.680141    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb107669-47a9-4a83-9268-3035b22408ce-bound-sa-token\") pod \"cert-manager-55b4d954d5-dxfff\" (UID: \"cb107669-47a9-4a83-9268-3035b22408ce\") " pod="cert-manager/cert-manager-55b4d954d5-dxfff"
Sep 15 07:27:21.680341 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:21.680192    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsxsz\" (UniqueName: \"kubernetes.io/projected/cb107669-47a9-4a83-9268-3035b22408ce-kube-api-access-hsxsz\") pod \"cert-manager-55b4d954d5-dxfff\" (UID: \"cb107669-47a9-4a83-9268-3035b22408ce\") " pod="cert-manager/cert-manager-55b4d954d5-dxfff"
Sep 15 07:27:21.688938 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:21.688906    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb107669-47a9-4a83-9268-3035b22408ce-bound-sa-token\") pod \"cert-manager-55b4d954d5-dxfff\" (UID: \"cb107669-47a9-4a83-9268-3035b22408ce\") " pod="cert-manager/cert-manager-55b4d954d5-dxfff"
Sep 15 07:27:21.689058 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:21.689047    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsxsz\" (UniqueName: \"kubernetes.io/projected/cb107669-47a9-4a83-9268-3035b22408ce-kube-api-access-hsxsz\") pod \"cert-manager-55b4d954d5-dxfff\" (UID: \"cb107669-47a9-4a83-9268-3035b22408ce\") " pod="cert-manager/cert-manager-55b4d954d5-dxfff"
Sep 15 07:27:21.791687 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:21.791605    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dxfff"
Sep 15 07:27:22.058406 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:22.058379    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dxfff"]
Sep 15 07:27:22.061402 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:27:22.061376    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb107669_47a9_4a83_9268_3035b22408ce.slice/crio-d3968b30404a7f601d1b8f0e8e23bce0b7bcc2e1dd282a57554d12f708efa4b3 WatchSource:0}: Error finding container d3968b30404a7f601d1b8f0e8e23bce0b7bcc2e1dd282a57554d12f708efa4b3: Status 404 returned error can't find the container with id d3968b30404a7f601d1b8f0e8e23bce0b7bcc2e1dd282a57554d12f708efa4b3
Sep 15 07:27:22.934276 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:22.934217    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dxfff" event={"ID":"cb107669-47a9-4a83-9268-3035b22408ce","Type":"ContainerStarted","Data":"d3968b30404a7f601d1b8f0e8e23bce0b7bcc2e1dd282a57554d12f708efa4b3"}
Sep 15 07:27:25.948960 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:25.948921    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dxfff" event={"ID":"cb107669-47a9-4a83-9268-3035b22408ce","Type":"ContainerStarted","Data":"65e90b88329b72c1afa8e2e7279ead2203f9f8b8bdb8bb0e138c3d4c31f022a8"}
Sep 15 07:27:25.963908 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:25.963858    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-dxfff" podStartSLOduration=1.847511862 podStartE2EDuration="4.963844341s" totalImagesPullingTime="3.116332479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:27:25.962848843 +0000 UTC m=+550.509418966" watchObservedRunningTime="2026-09-15 07:27:25.963844341 +0000 UTC m=+550.510414439"
Sep 15 07:27:45.365028 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.364986    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b9fc96c88-6m7jq"]
Sep 15 07:27:45.368897 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.368876    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7b9fc96c88-6m7jq"
Sep 15 07:27:45.371774 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.371749    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 15 07:27:45.372963 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.372942    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 15 07:27:45.373078 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.372973    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 15 07:27:45.373978 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.373954    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 15 07:27:45.374081 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.374021    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 15 07:27:45.374081 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.374027    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-skwqt\""
Sep 15 07:27:45.382488 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.382466    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b9fc96c88-6m7jq"]
Sep 15 07:27:45.480724 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.480688    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23099786-e37f-4df6-8d3e-38105d2a3f61-cert\") pod \"lws-controller-manager-7b9fc96c88-6m7jq\" (UID: \"23099786-e37f-4df6-8d3e-38105d2a3f61\") " pod="openshift-lws-operator/lws-controller-manager-7b9fc96c88-6m7jq"
Sep 15 07:27:45.480724 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.480724    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82fnk\" (UniqueName: \"kubernetes.io/projected/23099786-e37f-4df6-8d3e-38105d2a3f61-kube-api-access-82fnk\") pod \"lws-controller-manager-7b9fc96c88-6m7jq\" (UID: \"23099786-e37f-4df6-8d3e-38105d2a3f61\") " pod="openshift-lws-operator/lws-controller-manager-7b9fc96c88-6m7jq"
Sep 15 07:27:45.480943 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.480745    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/23099786-e37f-4df6-8d3e-38105d2a3f61-metrics-cert\") pod \"lws-controller-manager-7b9fc96c88-6m7jq\" (UID: \"23099786-e37f-4df6-8d3e-38105d2a3f61\") " pod="openshift-lws-operator/lws-controller-manager-7b9fc96c88-6m7jq"
Sep 15 07:27:45.480943 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.480771    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/23099786-e37f-4df6-8d3e-38105d2a3f61-manager-config\") pod \"lws-controller-manager-7b9fc96c88-6m7jq\" (UID: \"23099786-e37f-4df6-8d3e-38105d2a3f61\") " pod="openshift-lws-operator/lws-controller-manager-7b9fc96c88-6m7jq"
Sep 15 07:27:45.588004 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.587969    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23099786-e37f-4df6-8d3e-38105d2a3f61-cert\") pod \"lws-controller-manager-7b9fc96c88-6m7jq\" (UID: \"23099786-e37f-4df6-8d3e-38105d2a3f61\") " pod="openshift-lws-operator/lws-controller-manager-7b9fc96c88-6m7jq"
Sep 15 07:27:45.588004 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.588010    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82fnk\" (UniqueName: \"kubernetes.io/projected/23099786-e37f-4df6-8d3e-38105d2a3f61-kube-api-access-82fnk\") pod \"lws-controller-manager-7b9fc96c88-6m7jq\" (UID: \"23099786-e37f-4df6-8d3e-38105d2a3f61\") " pod="openshift-lws-operator/lws-controller-manager-7b9fc96c88-6m7jq"
Sep 15 07:27:45.588245 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.588034    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/23099786-e37f-4df6-8d3e-38105d2a3f61-metrics-cert\") pod \"lws-controller-manager-7b9fc96c88-6m7jq\" (UID: \"23099786-e37f-4df6-8d3e-38105d2a3f61\") " pod="openshift-lws-operator/lws-controller-manager-7b9fc96c88-6m7jq"
Sep 15 07:27:45.588245 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.588067    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/23099786-e37f-4df6-8d3e-38105d2a3f61-manager-config\") pod \"lws-controller-manager-7b9fc96c88-6m7jq\" (UID: \"23099786-e37f-4df6-8d3e-38105d2a3f61\") " pod="openshift-lws-operator/lws-controller-manager-7b9fc96c88-6m7jq"
Sep 15 07:27:45.588760 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.588721    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/23099786-e37f-4df6-8d3e-38105d2a3f61-manager-config\") pod \"lws-controller-manager-7b9fc96c88-6m7jq\" (UID: \"23099786-e37f-4df6-8d3e-38105d2a3f61\") " pod="openshift-lws-operator/lws-controller-manager-7b9fc96c88-6m7jq"
Sep 15 07:27:45.590630 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.590605    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23099786-e37f-4df6-8d3e-38105d2a3f61-cert\") pod \"lws-controller-manager-7b9fc96c88-6m7jq\" (UID: \"23099786-e37f-4df6-8d3e-38105d2a3f61\") " pod="openshift-lws-operator/lws-controller-manager-7b9fc96c88-6m7jq"
Sep 15 07:27:45.590720 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.590638    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/23099786-e37f-4df6-8d3e-38105d2a3f61-metrics-cert\") pod \"lws-controller-manager-7b9fc96c88-6m7jq\" (UID: \"23099786-e37f-4df6-8d3e-38105d2a3f61\") " pod="openshift-lws-operator/lws-controller-manager-7b9fc96c88-6m7jq"
Sep 15 07:27:45.596147 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.596128    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82fnk\" (UniqueName: \"kubernetes.io/projected/23099786-e37f-4df6-8d3e-38105d2a3f61-kube-api-access-82fnk\") pod \"lws-controller-manager-7b9fc96c88-6m7jq\" (UID: \"23099786-e37f-4df6-8d3e-38105d2a3f61\") " pod="openshift-lws-operator/lws-controller-manager-7b9fc96c88-6m7jq"
Sep 15 07:27:45.678396 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.678321    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7b9fc96c88-6m7jq"
Sep 15 07:27:45.949640 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:45.949615    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b9fc96c88-6m7jq"]
Sep 15 07:27:45.952340 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:27:45.952314    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23099786_e37f_4df6_8d3e_38105d2a3f61.slice/crio-875aabdc4549e71065f6b30d54c3e4bdd0a890aabc5727cd889a9f461340ba73 WatchSource:0}: Error finding container 875aabdc4549e71065f6b30d54c3e4bdd0a890aabc5727cd889a9f461340ba73: Status 404 returned error can't find the container with id 875aabdc4549e71065f6b30d54c3e4bdd0a890aabc5727cd889a9f461340ba73
Sep 15 07:27:46.030285 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.030234    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7b9fc96c88-6m7jq" event={"ID":"23099786-e37f-4df6-8d3e-38105d2a3f61","Type":"ContainerStarted","Data":"875aabdc4549e71065f6b30d54c3e4bdd0a890aabc5727cd889a9f461340ba73"}
Sep 15 07:27:46.353611 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.353567    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8"]
Sep 15 07:27:46.356685 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.356662    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8"
Sep 15 07:27:46.362729 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.361540    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 07:27:46.362729 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.361831    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 07:27:46.362729 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.362021    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qd2gg\""
Sep 15 07:27:46.362729 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.362542    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 07:27:46.363145 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.363127    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 07:27:46.378865 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.378840    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8"]
Sep 15 07:27:46.496907 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.496874    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c74ca08a-5740-4e74-b9b1-611d28dc143d-tmp\") pod \"opendatahub-operator-controller-manager-565d545f7b-7djq8\" (UID: \"c74ca08a-5740-4e74-b9b1-611d28dc143d\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8"
Sep 15 07:27:46.497075 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.496913    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c74ca08a-5740-4e74-b9b1-611d28dc143d-webhook-cert\") pod \"opendatahub-operator-controller-manager-565d545f7b-7djq8\" (UID: \"c74ca08a-5740-4e74-b9b1-611d28dc143d\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8"
Sep 15 07:27:46.497075 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.496940    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8nlh\" (UniqueName: \"kubernetes.io/projected/c74ca08a-5740-4e74-b9b1-611d28dc143d-kube-api-access-h8nlh\") pod \"opendatahub-operator-controller-manager-565d545f7b-7djq8\" (UID: \"c74ca08a-5740-4e74-b9b1-611d28dc143d\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8"
Sep 15 07:27:46.497075 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.497036    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c74ca08a-5740-4e74-b9b1-611d28dc143d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565d545f7b-7djq8\" (UID: \"c74ca08a-5740-4e74-b9b1-611d28dc143d\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8"
Sep 15 07:27:46.497216 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.497085    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c74ca08a-5740-4e74-b9b1-611d28dc143d-manifests\") pod \"opendatahub-operator-controller-manager-565d545f7b-7djq8\" (UID: \"c74ca08a-5740-4e74-b9b1-611d28dc143d\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8"
Sep 15 07:27:46.598039 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.598001    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c74ca08a-5740-4e74-b9b1-611d28dc143d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565d545f7b-7djq8\" (UID: \"c74ca08a-5740-4e74-b9b1-611d28dc143d\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8"
Sep 15 07:27:46.598211 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.598051    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c74ca08a-5740-4e74-b9b1-611d28dc143d-manifests\") pod \"opendatahub-operator-controller-manager-565d545f7b-7djq8\" (UID: \"c74ca08a-5740-4e74-b9b1-611d28dc143d\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8"
Sep 15 07:27:46.598211 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.598086    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c74ca08a-5740-4e74-b9b1-611d28dc143d-tmp\") pod \"opendatahub-operator-controller-manager-565d545f7b-7djq8\" (UID: \"c74ca08a-5740-4e74-b9b1-611d28dc143d\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8"
Sep 15 07:27:46.598211 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.598109    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c74ca08a-5740-4e74-b9b1-611d28dc143d-webhook-cert\") pod \"opendatahub-operator-controller-manager-565d545f7b-7djq8\" (UID: \"c74ca08a-5740-4e74-b9b1-611d28dc143d\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8"
Sep 15 07:27:46.598211 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.598132    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8nlh\" (UniqueName: \"kubernetes.io/projected/c74ca08a-5740-4e74-b9b1-611d28dc143d-kube-api-access-h8nlh\") pod \"opendatahub-operator-controller-manager-565d545f7b-7djq8\" (UID: \"c74ca08a-5740-4e74-b9b1-611d28dc143d\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8"
Sep 15 07:27:46.598589 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.598568    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c74ca08a-5740-4e74-b9b1-611d28dc143d-manifests\") pod \"opendatahub-operator-controller-manager-565d545f7b-7djq8\" (UID: \"c74ca08a-5740-4e74-b9b1-611d28dc143d\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8"
Sep 15 07:27:46.600400 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.600373    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c74ca08a-5740-4e74-b9b1-611d28dc143d-tmp\") pod \"opendatahub-operator-controller-manager-565d545f7b-7djq8\" (UID: \"c74ca08a-5740-4e74-b9b1-611d28dc143d\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8"
Sep 15 07:27:46.600550 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.600533    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c74ca08a-5740-4e74-b9b1-611d28dc143d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565d545f7b-7djq8\" (UID: \"c74ca08a-5740-4e74-b9b1-611d28dc143d\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8"
Sep 15 07:27:46.600599 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.600583    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c74ca08a-5740-4e74-b9b1-611d28dc143d-webhook-cert\") pod \"opendatahub-operator-controller-manager-565d545f7b-7djq8\" (UID: \"c74ca08a-5740-4e74-b9b1-611d28dc143d\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8"
Sep 15 07:27:46.606608 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.606561    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8nlh\" (UniqueName: \"kubernetes.io/projected/c74ca08a-5740-4e74-b9b1-611d28dc143d-kube-api-access-h8nlh\") pod \"opendatahub-operator-controller-manager-565d545f7b-7djq8\" (UID: \"c74ca08a-5740-4e74-b9b1-611d28dc143d\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8"
Sep 15 07:27:46.670628 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.670603    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8"
Sep 15 07:27:46.947889 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:46.947862    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8"]
Sep 15 07:27:46.952979 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:27:46.952950    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74ca08a_5740_4e74_b9b1_611d28dc143d.slice/crio-9578c2560df267399b59086bdf58a9a9d565f1be25f3cc8727597070a1c45e9f WatchSource:0}: Error finding container 9578c2560df267399b59086bdf58a9a9d565f1be25f3cc8727597070a1c45e9f: Status 404 returned error can't find the container with id 9578c2560df267399b59086bdf58a9a9d565f1be25f3cc8727597070a1c45e9f
Sep 15 07:27:47.038955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:47.038913    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8" event={"ID":"c74ca08a-5740-4e74-b9b1-611d28dc143d","Type":"ContainerStarted","Data":"9578c2560df267399b59086bdf58a9a9d565f1be25f3cc8727597070a1c45e9f"}
Sep 15 07:27:49.051686 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:49.051515    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7b9fc96c88-6m7jq" event={"ID":"23099786-e37f-4df6-8d3e-38105d2a3f61","Type":"ContainerStarted","Data":"fcec6c289dba3db006c8ac0471de013075a4c2c4b588e6d1a310d64c60e29c0c"}
Sep 15 07:27:49.051686 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:49.051613    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7b9fc96c88-6m7jq"
Sep 15 07:27:49.107703 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:49.107646    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7b9fc96c88-6m7jq" podStartSLOduration=1.48774933 podStartE2EDuration="4.107621501s" totalImagesPullingTime="2.619872171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:27:49.104866947 +0000 UTC m=+573.651437041" watchObservedRunningTime="2026-09-15 07:27:49.107621501 +0000 UTC m=+573.654191601"
Sep 15 07:27:51.061629 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:51.061596    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8" event={"ID":"c74ca08a-5740-4e74-b9b1-611d28dc143d","Type":"ContainerStarted","Data":"eabf25cb19b39a949436f92439f8431bc3d373960b7fed4993ef5b9955fea9a3"}
Sep 15 07:27:53.072105 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:53.072072    2571 generic.go:356] "Generic (PLEG): container finished" podID="c74ca08a-5740-4e74-b9b1-611d28dc143d" containerID="eabf25cb19b39a949436f92439f8431bc3d373960b7fed4993ef5b9955fea9a3" exitCode=0
Sep 15 07:27:53.072510 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:53.072130    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8" event={"ID":"c74ca08a-5740-4e74-b9b1-611d28dc143d","Type":"ContainerDied","Data":"eabf25cb19b39a949436f92439f8431bc3d373960b7fed4993ef5b9955fea9a3"}
Sep 15 07:27:54.078733 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:54.078696    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8" event={"ID":"c74ca08a-5740-4e74-b9b1-611d28dc143d","Type":"ContainerStarted","Data":"37f874b093c4a635e9dc767120f51c9ca5ac67039d97204d1de04c62331e1143"}
Sep 15 07:27:54.079141 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:54.079024    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8"
Sep 15 07:27:54.102627 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:27:54.102577    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8" podStartSLOduration=2.659649183 podStartE2EDuration="8.102562005s" totalImagesPullingTime="4.356132056s" totalInitContainerRuntime="1.086780766s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:27:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:27:54.100804815 +0000 UTC m=+578.647374959" watchObservedRunningTime="2026-09-15 07:27:54.102562005 +0000 UTC m=+578.649132103"
Sep 15 07:28:00.058488 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:00.058450    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7b9fc96c88-6m7jq"
Sep 15 07:28:05.085175 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:05.085144    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-7djq8"
Sep 15 07:28:08.273813 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:08.273777    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-77b7bc76bc-mrxgc"]
Sep 15 07:28:08.279691 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:08.279669    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-mrxgc"
Sep 15 07:28:08.282624 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:08.282601    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 07:28:08.282863 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:08.282842    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 07:28:08.282968 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:08.282896    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 15 07:28:08.282968 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:08.282897    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jnm2p\""
Sep 15 07:28:08.283107 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:08.283093    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 15 07:28:08.292880 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:08.292852    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77b7bc76bc-mrxgc"]
Sep 15 07:28:08.309310 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:08.309284    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02c7ea90-3665-4228-9470-871cebea1479-tmp\") pod \"kube-auth-proxy-77b7bc76bc-mrxgc\" (UID: \"02c7ea90-3665-4228-9470-871cebea1479\") " pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-mrxgc"
Sep 15 07:28:08.309447 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:08.309326    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2stcv\" (UniqueName: \"kubernetes.io/projected/02c7ea90-3665-4228-9470-871cebea1479-kube-api-access-2stcv\") pod \"kube-auth-proxy-77b7bc76bc-mrxgc\" (UID: \"02c7ea90-3665-4228-9470-871cebea1479\") " pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-mrxgc"
Sep 15 07:28:08.309447 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:08.309413    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02c7ea90-3665-4228-9470-871cebea1479-tls-certs\") pod \"kube-auth-proxy-77b7bc76bc-mrxgc\" (UID: \"02c7ea90-3665-4228-9470-871cebea1479\") " pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-mrxgc"
Sep 15 07:28:08.410783 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:08.410746    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02c7ea90-3665-4228-9470-871cebea1479-tls-certs\") pod \"kube-auth-proxy-77b7bc76bc-mrxgc\" (UID: \"02c7ea90-3665-4228-9470-871cebea1479\") " pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-mrxgc"
Sep 15 07:28:08.410994 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:08.410843    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02c7ea90-3665-4228-9470-871cebea1479-tmp\") pod \"kube-auth-proxy-77b7bc76bc-mrxgc\" (UID: \"02c7ea90-3665-4228-9470-871cebea1479\") " pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-mrxgc"
Sep 15 07:28:08.410994 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:08.410931    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2stcv\" (UniqueName: \"kubernetes.io/projected/02c7ea90-3665-4228-9470-871cebea1479-kube-api-access-2stcv\") pod \"kube-auth-proxy-77b7bc76bc-mrxgc\" (UID: \"02c7ea90-3665-4228-9470-871cebea1479\") " pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-mrxgc"
Sep 15 07:28:08.413036 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:08.413006    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02c7ea90-3665-4228-9470-871cebea1479-tmp\") pod \"kube-auth-proxy-77b7bc76bc-mrxgc\" (UID: \"02c7ea90-3665-4228-9470-871cebea1479\") " pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-mrxgc"
Sep 15 07:28:08.413233 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:08.413214    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02c7ea90-3665-4228-9470-871cebea1479-tls-certs\") pod \"kube-auth-proxy-77b7bc76bc-mrxgc\" (UID: \"02c7ea90-3665-4228-9470-871cebea1479\") " pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-mrxgc"
Sep 15 07:28:08.427810 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:08.427785    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2stcv\" (UniqueName: \"kubernetes.io/projected/02c7ea90-3665-4228-9470-871cebea1479-kube-api-access-2stcv\") pod \"kube-auth-proxy-77b7bc76bc-mrxgc\" (UID: \"02c7ea90-3665-4228-9470-871cebea1479\") " pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-mrxgc"
Sep 15 07:28:08.589649 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:08.589551    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-mrxgc"
Sep 15 07:28:08.863474 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:08.863447    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77b7bc76bc-mrxgc"]
Sep 15 07:28:08.865960 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:28:08.865939    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c7ea90_3665_4228_9470_871cebea1479.slice/crio-4969349c0b0c02a39fbe64a918d27fbe88c03c97c0a2a2fc62879948d48ddd24 WatchSource:0}: Error finding container 4969349c0b0c02a39fbe64a918d27fbe88c03c97c0a2a2fc62879948d48ddd24: Status 404 returned error can't find the container with id 4969349c0b0c02a39fbe64a918d27fbe88c03c97c0a2a2fc62879948d48ddd24
Sep 15 07:28:09.145373 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:09.145288    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-mrxgc" event={"ID":"02c7ea90-3665-4228-9470-871cebea1479","Type":"ContainerStarted","Data":"4969349c0b0c02a39fbe64a918d27fbe88c03c97c0a2a2fc62879948d48ddd24"}
Sep 15 07:28:12.162655 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:12.162571    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-mrxgc" event={"ID":"02c7ea90-3665-4228-9470-871cebea1479","Type":"ContainerStarted","Data":"321cbe159e754854f42d2083e7d70efc44986ed217adb86ce0a8575b14032222"}
Sep 15 07:28:12.180680 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:28:12.180630    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-mrxgc" podStartSLOduration=1.313415317 podStartE2EDuration="4.18061263s" totalImagesPullingTime="2.867197313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:28:12.178943204 +0000 UTC m=+596.725513322" watchObservedRunningTime="2026-09-15 07:28:12.18061263 +0000 UTC m=+596.727182787"
Sep 15 07:29:47.851640 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:47.851605    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-6gp2q"]
Sep 15 07:29:47.853963 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:47.853945    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6gp2q"
Sep 15 07:29:47.856394 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:47.856375    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 15 07:29:47.856484 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:47.856411    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 07:29:47.856525 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:47.856485    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rkdv7\""
Sep 15 07:29:47.857305 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:47.857285    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 15 07:29:47.857399 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:47.857295    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 07:29:47.864100 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:47.864078    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-6gp2q"]
Sep 15 07:29:47.884639 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:47.884605    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p97bg\" (UniqueName: \"kubernetes.io/projected/1012ba59-824f-42bb-bd39-3c2ebac8e491-kube-api-access-p97bg\") pod \"kuadrant-console-plugin-fbdd4f445-6gp2q\" (UID: \"1012ba59-824f-42bb-bd39-3c2ebac8e491\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6gp2q"
Sep 15 07:29:47.884781 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:47.884654    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1012ba59-824f-42bb-bd39-3c2ebac8e491-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-6gp2q\" (UID: \"1012ba59-824f-42bb-bd39-3c2ebac8e491\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6gp2q"
Sep 15 07:29:47.884781 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:47.884741    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1012ba59-824f-42bb-bd39-3c2ebac8e491-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-6gp2q\" (UID: \"1012ba59-824f-42bb-bd39-3c2ebac8e491\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6gp2q"
Sep 15 07:29:47.985600 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:47.985557    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1012ba59-824f-42bb-bd39-3c2ebac8e491-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-6gp2q\" (UID: \"1012ba59-824f-42bb-bd39-3c2ebac8e491\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6gp2q"
Sep 15 07:29:47.985780 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:47.985624    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1012ba59-824f-42bb-bd39-3c2ebac8e491-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-6gp2q\" (UID: \"1012ba59-824f-42bb-bd39-3c2ebac8e491\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6gp2q"
Sep 15 07:29:47.985780 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:47.985668    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p97bg\" (UniqueName: \"kubernetes.io/projected/1012ba59-824f-42bb-bd39-3c2ebac8e491-kube-api-access-p97bg\") pod \"kuadrant-console-plugin-fbdd4f445-6gp2q\" (UID: \"1012ba59-824f-42bb-bd39-3c2ebac8e491\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6gp2q"
Sep 15 07:29:47.985867 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:29:47.985769    2571 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 15 07:29:47.985867 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:29:47.985859    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1012ba59-824f-42bb-bd39-3c2ebac8e491-plugin-serving-cert podName:1012ba59-824f-42bb-bd39-3c2ebac8e491 nodeName:}" failed. No retries permitted until 2026-09-15 07:29:48.485837294 +0000 UTC m=+693.032407387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/1012ba59-824f-42bb-bd39-3c2ebac8e491-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-6gp2q" (UID: "1012ba59-824f-42bb-bd39-3c2ebac8e491") : secret "plugin-serving-cert" not found
Sep 15 07:29:47.986326 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:47.986307    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1012ba59-824f-42bb-bd39-3c2ebac8e491-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-6gp2q\" (UID: \"1012ba59-824f-42bb-bd39-3c2ebac8e491\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6gp2q"
Sep 15 07:29:47.993905 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:47.993881    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p97bg\" (UniqueName: \"kubernetes.io/projected/1012ba59-824f-42bb-bd39-3c2ebac8e491-kube-api-access-p97bg\") pod \"kuadrant-console-plugin-fbdd4f445-6gp2q\" (UID: \"1012ba59-824f-42bb-bd39-3c2ebac8e491\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6gp2q"
Sep 15 07:29:48.489026 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:48.488985    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1012ba59-824f-42bb-bd39-3c2ebac8e491-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-6gp2q\" (UID: \"1012ba59-824f-42bb-bd39-3c2ebac8e491\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6gp2q"
Sep 15 07:29:48.491521 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:48.491498    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1012ba59-824f-42bb-bd39-3c2ebac8e491-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-6gp2q\" (UID: \"1012ba59-824f-42bb-bd39-3c2ebac8e491\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6gp2q"
Sep 15 07:29:48.763771 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:48.763673    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6gp2q"
Sep 15 07:29:49.031121 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:49.030992    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-6gp2q"]
Sep 15 07:29:49.033835 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:29:49.033807    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1012ba59_824f_42bb_bd39_3c2ebac8e491.slice/crio-6054352944f3a361365203cf03b53f2b8c1d26edc75cec50119877299a3f8167 WatchSource:0}: Error finding container 6054352944f3a361365203cf03b53f2b8c1d26edc75cec50119877299a3f8167: Status 404 returned error can't find the container with id 6054352944f3a361365203cf03b53f2b8c1d26edc75cec50119877299a3f8167
Sep 15 07:29:49.035147 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:49.035125    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 07:29:49.617277 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:49.617189    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6gp2q" event={"ID":"1012ba59-824f-42bb-bd39-3c2ebac8e491","Type":"ContainerStarted","Data":"6054352944f3a361365203cf03b53f2b8c1d26edc75cec50119877299a3f8167"}
Sep 15 07:29:53.639332 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:53.639299    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6gp2q" event={"ID":"1012ba59-824f-42bb-bd39-3c2ebac8e491","Type":"ContainerStarted","Data":"ac4f255c9ace95ac4a9bcf448f2faf4df67eb2ccd6b653deab38ad6370fc533d"}
Sep 15 07:29:53.655034 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:53.654987    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6gp2q" podStartSLOduration=2.537254012 podStartE2EDuration="6.654972402s" totalImagesPullingTime="4.11771839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:29:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:29:53.65336423 +0000 UTC m=+698.199934322" watchObservedRunningTime="2026-09-15 07:29:53.654972402 +0000 UTC m=+698.201542501"
Sep 15 07:29:58.808162 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:58.808128    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-2zv8v"]
Sep 15 07:29:58.811711 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:58.811693    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2zv8v"
Sep 15 07:29:58.814280 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:58.814246    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-nn6rf\""
Sep 15 07:29:58.821956 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:58.821929    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-2zv8v"]
Sep 15 07:29:58.881358 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:58.881317    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq6vp\" (UniqueName: \"kubernetes.io/projected/21d46833-6a60-46fb-8498-87a141821243-kube-api-access-xq6vp\") pod \"limitador-operator-controller-manager-74495c69f8-2zv8v\" (UID: \"21d46833-6a60-46fb-8498-87a141821243\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2zv8v"
Sep 15 07:29:58.982840 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:58.982803    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq6vp\" (UniqueName: \"kubernetes.io/projected/21d46833-6a60-46fb-8498-87a141821243-kube-api-access-xq6vp\") pod \"limitador-operator-controller-manager-74495c69f8-2zv8v\" (UID: \"21d46833-6a60-46fb-8498-87a141821243\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2zv8v"
Sep 15 07:29:58.995382 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:58.995351    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq6vp\" (UniqueName: \"kubernetes.io/projected/21d46833-6a60-46fb-8498-87a141821243-kube-api-access-xq6vp\") pod \"limitador-operator-controller-manager-74495c69f8-2zv8v\" (UID: \"21d46833-6a60-46fb-8498-87a141821243\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2zv8v"
Sep 15 07:29:59.124553 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:59.124468    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2zv8v"
Sep 15 07:29:59.394205 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:59.394184    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-2zv8v"]
Sep 15 07:29:59.397210 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:29:59.397183    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d46833_6a60_46fb_8498_87a141821243.slice/crio-96b3f3a2b0dc0f6e3637d9db9a71112a6daefeced0ad974e4d4e05205c3a85ac WatchSource:0}: Error finding container 96b3f3a2b0dc0f6e3637d9db9a71112a6daefeced0ad974e4d4e05205c3a85ac: Status 404 returned error can't find the container with id 96b3f3a2b0dc0f6e3637d9db9a71112a6daefeced0ad974e4d4e05205c3a85ac
Sep 15 07:29:59.666769 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:29:59.666685    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2zv8v" event={"ID":"21d46833-6a60-46fb-8498-87a141821243","Type":"ContainerStarted","Data":"96b3f3a2b0dc0f6e3637d9db9a71112a6daefeced0ad974e4d4e05205c3a85ac"}
Sep 15 07:30:01.678991 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:01.678937    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2zv8v" event={"ID":"21d46833-6a60-46fb-8498-87a141821243","Type":"ContainerStarted","Data":"2c5c8623d47e95c3576a0aaaf28e54c2a1fed671945fce8d84e8bce5ece4e37a"}
Sep 15 07:30:01.679389 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:01.679025    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2zv8v"
Sep 15 07:30:01.700769 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:01.700721    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2zv8v" podStartSLOduration=1.600838217 podStartE2EDuration="3.70070396s" totalImagesPullingTime="2.099865743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:29:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:30:01.699696796 +0000 UTC m=+706.246266895" watchObservedRunningTime="2026-09-15 07:30:01.70070396 +0000 UTC m=+706.247274059"
Sep 15 07:30:12.686279 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:12.686182    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2zv8v"
Sep 15 07:30:34.955041 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:34.955006    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-kdhmw"]
Sep 15 07:30:34.986235 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:34.986206    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-kdhmw"]
Sep 15 07:30:34.986408 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:34.986320    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-kdhmw"
Sep 15 07:30:34.988594 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:34.988572    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rc8pz\""
Sep 15 07:30:35.104701 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:35.104666    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcfvc\" (UniqueName: \"kubernetes.io/projected/d57ebaa5-2472-44e8-a379-ad3b4426ef56-kube-api-access-lcfvc\") pod \"authorino-7bcfd8f9cd-kdhmw\" (UID: \"d57ebaa5-2472-44e8-a379-ad3b4426ef56\") " pod="kuadrant-system/authorino-7bcfd8f9cd-kdhmw"
Sep 15 07:30:35.205710 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:35.205621    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcfvc\" (UniqueName: \"kubernetes.io/projected/d57ebaa5-2472-44e8-a379-ad3b4426ef56-kube-api-access-lcfvc\") pod \"authorino-7bcfd8f9cd-kdhmw\" (UID: \"d57ebaa5-2472-44e8-a379-ad3b4426ef56\") " pod="kuadrant-system/authorino-7bcfd8f9cd-kdhmw"
Sep 15 07:30:35.213822 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:35.213792    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcfvc\" (UniqueName: \"kubernetes.io/projected/d57ebaa5-2472-44e8-a379-ad3b4426ef56-kube-api-access-lcfvc\") pod \"authorino-7bcfd8f9cd-kdhmw\" (UID: \"d57ebaa5-2472-44e8-a379-ad3b4426ef56\") " pod="kuadrant-system/authorino-7bcfd8f9cd-kdhmw"
Sep 15 07:30:35.295368 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:35.295334    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-kdhmw"
Sep 15 07:30:35.560992 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:35.560970    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-kdhmw"]
Sep 15 07:30:35.562904 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:30:35.562877    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd57ebaa5_2472_44e8_a379_ad3b4426ef56.slice/crio-f4d5500255c26c40c0fbd736e5db9b6e5fa7d593d98276ce094ef61b4025fcf3 WatchSource:0}: Error finding container f4d5500255c26c40c0fbd736e5db9b6e5fa7d593d98276ce094ef61b4025fcf3: Status 404 returned error can't find the container with id f4d5500255c26c40c0fbd736e5db9b6e5fa7d593d98276ce094ef61b4025fcf3
Sep 15 07:30:35.837606 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:35.837520    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-kdhmw" event={"ID":"d57ebaa5-2472-44e8-a379-ad3b4426ef56","Type":"ContainerStarted","Data":"f4d5500255c26c40c0fbd736e5db9b6e5fa7d593d98276ce094ef61b4025fcf3"}
Sep 15 07:30:37.855474 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:37.855434    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-kdhmw" event={"ID":"d57ebaa5-2472-44e8-a379-ad3b4426ef56","Type":"ContainerStarted","Data":"98d518631f5c2dc95e5018a699c8153cb2b1148533f60e1337d7ebf71250e1fd"}
Sep 15 07:30:37.870860 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:37.870805    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-kdhmw" podStartSLOduration=2.032827841 podStartE2EDuration="3.870790378s" totalImagesPullingTime="1.837962537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:30:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:30:37.869019519 +0000 UTC m=+742.415589620" watchObservedRunningTime="2026-09-15 07:30:37.870790378 +0000 UTC m=+742.417360477"
Sep 15 07:30:38.203564 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:38.203531    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-kdhmw"]
Sep 15 07:30:39.864888 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:39.864846    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-kdhmw" podUID="d57ebaa5-2472-44e8-a379-ad3b4426ef56" containerName="authorino" containerID="cri-o://98d518631f5c2dc95e5018a699c8153cb2b1148533f60e1337d7ebf71250e1fd" gracePeriod=30
Sep 15 07:30:40.106157 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:40.106130    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-kdhmw"
Sep 15 07:30:40.152352 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:40.152246    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcfvc\" (UniqueName: \"kubernetes.io/projected/d57ebaa5-2472-44e8-a379-ad3b4426ef56-kube-api-access-lcfvc\") pod \"d57ebaa5-2472-44e8-a379-ad3b4426ef56\" (UID: \"d57ebaa5-2472-44e8-a379-ad3b4426ef56\") "
Sep 15 07:30:40.154480 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:40.154448    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d57ebaa5-2472-44e8-a379-ad3b4426ef56-kube-api-access-lcfvc" (OuterVolumeSpecName: "kube-api-access-lcfvc") pod "d57ebaa5-2472-44e8-a379-ad3b4426ef56" (UID: "d57ebaa5-2472-44e8-a379-ad3b4426ef56"). InnerVolumeSpecName "kube-api-access-lcfvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:30:40.253132 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:40.253104    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcfvc\" (UniqueName: \"kubernetes.io/projected/d57ebaa5-2472-44e8-a379-ad3b4426ef56-kube-api-access-lcfvc\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:30:40.870229 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:40.870191    2571 generic.go:356] "Generic (PLEG): container finished" podID="d57ebaa5-2472-44e8-a379-ad3b4426ef56" containerID="98d518631f5c2dc95e5018a699c8153cb2b1148533f60e1337d7ebf71250e1fd" exitCode=0
Sep 15 07:30:40.870842 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:40.870236    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-kdhmw"
Sep 15 07:30:40.870842 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:40.870246    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-kdhmw" event={"ID":"d57ebaa5-2472-44e8-a379-ad3b4426ef56","Type":"ContainerDied","Data":"98d518631f5c2dc95e5018a699c8153cb2b1148533f60e1337d7ebf71250e1fd"}
Sep 15 07:30:40.870842 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:40.870308    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-kdhmw" event={"ID":"d57ebaa5-2472-44e8-a379-ad3b4426ef56","Type":"ContainerDied","Data":"f4d5500255c26c40c0fbd736e5db9b6e5fa7d593d98276ce094ef61b4025fcf3"}
Sep 15 07:30:40.870842 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:40.870328    2571 scope.go:117] "RemoveContainer" containerID="98d518631f5c2dc95e5018a699c8153cb2b1148533f60e1337d7ebf71250e1fd"
Sep 15 07:30:40.878684 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:40.878665    2571 scope.go:117] "RemoveContainer" containerID="98d518631f5c2dc95e5018a699c8153cb2b1148533f60e1337d7ebf71250e1fd"
Sep 15 07:30:40.878913 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:30:40.878895    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98d518631f5c2dc95e5018a699c8153cb2b1148533f60e1337d7ebf71250e1fd\": container with ID starting with 98d518631f5c2dc95e5018a699c8153cb2b1148533f60e1337d7ebf71250e1fd not found: ID does not exist" containerID="98d518631f5c2dc95e5018a699c8153cb2b1148533f60e1337d7ebf71250e1fd"
Sep 15 07:30:40.878956 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:40.878923    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d518631f5c2dc95e5018a699c8153cb2b1148533f60e1337d7ebf71250e1fd"} err="failed to get container status \"98d518631f5c2dc95e5018a699c8153cb2b1148533f60e1337d7ebf71250e1fd\": rpc error: code = NotFound desc = could not find container \"98d518631f5c2dc95e5018a699c8153cb2b1148533f60e1337d7ebf71250e1fd\": container with ID starting with 98d518631f5c2dc95e5018a699c8153cb2b1148533f60e1337d7ebf71250e1fd not found: ID does not exist"
Sep 15 07:30:40.892465 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:40.892436    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-kdhmw"]
Sep 15 07:30:40.894102 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:40.894079    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-kdhmw"]
Sep 15 07:30:41.908470 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:30:41.908438    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d57ebaa5-2472-44e8-a379-ad3b4426ef56" path="/var/lib/kubelet/pods/d57ebaa5-2472-44e8-a379-ad3b4426ef56/volumes"
Sep 15 07:31:02.809983 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:02.809950    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-rv5f5"]
Sep 15 07:31:02.810494 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:02.810456    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d57ebaa5-2472-44e8-a379-ad3b4426ef56" containerName="authorino"
Sep 15 07:31:02.810494 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:02.810471    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d57ebaa5-2472-44e8-a379-ad3b4426ef56" containerName="authorino"
Sep 15 07:31:02.810578 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:02.810556    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d57ebaa5-2472-44e8-a379-ad3b4426ef56" containerName="authorino"
Sep 15 07:31:02.813647 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:02.813631    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rv5f5"
Sep 15 07:31:02.815904 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:02.815885    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rc8pz\""
Sep 15 07:31:02.821873 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:02.821842    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-rv5f5"]
Sep 15 07:31:02.851669 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:02.851635    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfsjx\" (UniqueName: \"kubernetes.io/projected/795c2dc9-9c21-4070-8533-47e31c9c6770-kube-api-access-jfsjx\") pod \"authorino-557f876f69-rv5f5\" (UID: \"795c2dc9-9c21-4070-8533-47e31c9c6770\") " pod="kuadrant-system/authorino-557f876f69-rv5f5"
Sep 15 07:31:02.859164 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:02.859117    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-rv5f5"]
Sep 15 07:31:02.864363 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:31:02.864320    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-jfsjx], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-rv5f5" podUID="795c2dc9-9c21-4070-8533-47e31c9c6770"
Sep 15 07:31:02.884677 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:02.884644    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-7v7h7"]
Sep 15 07:31:02.888010 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:02.887990    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-7v7h7"
Sep 15 07:31:02.890487 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:02.890468    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 15 07:31:02.895052 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:02.895025    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-7v7h7"]
Sep 15 07:31:02.952980 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:02.952938    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616-tls-cert\") pod \"authorino-6d4dfc649f-7v7h7\" (UID: \"4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616\") " pod="kuadrant-system/authorino-6d4dfc649f-7v7h7"
Sep 15 07:31:02.953148 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:02.953044    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfsjx\" (UniqueName: \"kubernetes.io/projected/795c2dc9-9c21-4070-8533-47e31c9c6770-kube-api-access-jfsjx\") pod \"authorino-557f876f69-rv5f5\" (UID: \"795c2dc9-9c21-4070-8533-47e31c9c6770\") " pod="kuadrant-system/authorino-557f876f69-rv5f5"
Sep 15 07:31:02.953148 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:02.953061    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vgm9\" (UniqueName: \"kubernetes.io/projected/4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616-kube-api-access-2vgm9\") pod \"authorino-6d4dfc649f-7v7h7\" (UID: \"4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616\") " pod="kuadrant-system/authorino-6d4dfc649f-7v7h7"
Sep 15 07:31:02.960748 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:02.960720    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfsjx\" (UniqueName: \"kubernetes.io/projected/795c2dc9-9c21-4070-8533-47e31c9c6770-kube-api-access-jfsjx\") pod \"authorino-557f876f69-rv5f5\" (UID: \"795c2dc9-9c21-4070-8533-47e31c9c6770\") " pod="kuadrant-system/authorino-557f876f69-rv5f5"
Sep 15 07:31:02.970797 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:02.970773    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rv5f5"
Sep 15 07:31:02.977844 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:02.977822    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rv5f5"
Sep 15 07:31:03.042155 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.042115    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-7v7h7"]
Sep 15 07:31:03.042433 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:31:03.042411    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-2vgm9 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-7v7h7" podUID="4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616"
Sep 15 07:31:03.053575 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.053551    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfsjx\" (UniqueName: \"kubernetes.io/projected/795c2dc9-9c21-4070-8533-47e31c9c6770-kube-api-access-jfsjx\") pod \"795c2dc9-9c21-4070-8533-47e31c9c6770\" (UID: \"795c2dc9-9c21-4070-8533-47e31c9c6770\") "
Sep 15 07:31:03.053710 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.053694    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vgm9\" (UniqueName: \"kubernetes.io/projected/4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616-kube-api-access-2vgm9\") pod \"authorino-6d4dfc649f-7v7h7\" (UID: \"4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616\") " pod="kuadrant-system/authorino-6d4dfc649f-7v7h7"
Sep 15 07:31:03.053756 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.053737    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616-tls-cert\") pod \"authorino-6d4dfc649f-7v7h7\" (UID: \"4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616\") " pod="kuadrant-system/authorino-6d4dfc649f-7v7h7"
Sep 15 07:31:03.055696 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.055665    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795c2dc9-9c21-4070-8533-47e31c9c6770-kube-api-access-jfsjx" (OuterVolumeSpecName: "kube-api-access-jfsjx") pod "795c2dc9-9c21-4070-8533-47e31c9c6770" (UID: "795c2dc9-9c21-4070-8533-47e31c9c6770"). InnerVolumeSpecName "kube-api-access-jfsjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:31:03.056047 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.056028    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616-tls-cert\") pod \"authorino-6d4dfc649f-7v7h7\" (UID: \"4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616\") " pod="kuadrant-system/authorino-6d4dfc649f-7v7h7"
Sep 15 07:31:03.065519 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.065497    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vgm9\" (UniqueName: \"kubernetes.io/projected/4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616-kube-api-access-2vgm9\") pod \"authorino-6d4dfc649f-7v7h7\" (UID: \"4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616\") " pod="kuadrant-system/authorino-6d4dfc649f-7v7h7"
Sep 15 07:31:03.071878 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.071850    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-855d77dd88-bmzpn"]
Sep 15 07:31:03.075187 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.075166    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-855d77dd88-bmzpn"
Sep 15 07:31:03.082403 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.082379    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-855d77dd88-bmzpn"]
Sep 15 07:31:03.154544 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.154511    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mphb\" (UniqueName: \"kubernetes.io/projected/667b93e7-d1d6-4ac0-92c4-e30040d80805-kube-api-access-2mphb\") pod \"authorino-855d77dd88-bmzpn\" (UID: \"667b93e7-d1d6-4ac0-92c4-e30040d80805\") " pod="kuadrant-system/authorino-855d77dd88-bmzpn"
Sep 15 07:31:03.154544 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.154560    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/667b93e7-d1d6-4ac0-92c4-e30040d80805-tls-cert\") pod \"authorino-855d77dd88-bmzpn\" (UID: \"667b93e7-d1d6-4ac0-92c4-e30040d80805\") " pod="kuadrant-system/authorino-855d77dd88-bmzpn"
Sep 15 07:31:03.154747 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.154639    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfsjx\" (UniqueName: \"kubernetes.io/projected/795c2dc9-9c21-4070-8533-47e31c9c6770-kube-api-access-jfsjx\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:31:03.255428 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.255389    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mphb\" (UniqueName: \"kubernetes.io/projected/667b93e7-d1d6-4ac0-92c4-e30040d80805-kube-api-access-2mphb\") pod \"authorino-855d77dd88-bmzpn\" (UID: \"667b93e7-d1d6-4ac0-92c4-e30040d80805\") " pod="kuadrant-system/authorino-855d77dd88-bmzpn"
Sep 15 07:31:03.255597 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.255442    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/667b93e7-d1d6-4ac0-92c4-e30040d80805-tls-cert\") pod \"authorino-855d77dd88-bmzpn\" (UID: \"667b93e7-d1d6-4ac0-92c4-e30040d80805\") " pod="kuadrant-system/authorino-855d77dd88-bmzpn"
Sep 15 07:31:03.257781 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.257761    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/667b93e7-d1d6-4ac0-92c4-e30040d80805-tls-cert\") pod \"authorino-855d77dd88-bmzpn\" (UID: \"667b93e7-d1d6-4ac0-92c4-e30040d80805\") " pod="kuadrant-system/authorino-855d77dd88-bmzpn"
Sep 15 07:31:03.263754 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.263729    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mphb\" (UniqueName: \"kubernetes.io/projected/667b93e7-d1d6-4ac0-92c4-e30040d80805-kube-api-access-2mphb\") pod \"authorino-855d77dd88-bmzpn\" (UID: \"667b93e7-d1d6-4ac0-92c4-e30040d80805\") " pod="kuadrant-system/authorino-855d77dd88-bmzpn"
Sep 15 07:31:03.388245 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.388161    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-855d77dd88-bmzpn"
Sep 15 07:31:03.655724 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.655652    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-855d77dd88-bmzpn"]
Sep 15 07:31:03.658479 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:31:03.658453    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod667b93e7_d1d6_4ac0_92c4_e30040d80805.slice/crio-fc0489eddd9bfc2b2b1bf1bfd030b73011f72381ce29ed7c659f919e78238aa4 WatchSource:0}: Error finding container fc0489eddd9bfc2b2b1bf1bfd030b73011f72381ce29ed7c659f919e78238aa4: Status 404 returned error can't find the container with id fc0489eddd9bfc2b2b1bf1bfd030b73011f72381ce29ed7c659f919e78238aa4
Sep 15 07:31:03.976764 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.976728    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-855d77dd88-bmzpn" event={"ID":"667b93e7-d1d6-4ac0-92c4-e30040d80805","Type":"ContainerStarted","Data":"fc0489eddd9bfc2b2b1bf1bfd030b73011f72381ce29ed7c659f919e78238aa4"}
Sep 15 07:31:03.977133 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.976740    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-7v7h7"
Sep 15 07:31:03.977133 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.976750    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rv5f5"
Sep 15 07:31:03.985782 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:03.985759    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-7v7h7"
Sep 15 07:31:04.013169 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:04.013137    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-rv5f5"]
Sep 15 07:31:04.017028 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:04.017002    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-rv5f5"]
Sep 15 07:31:04.061919 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:04.061889    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vgm9\" (UniqueName: \"kubernetes.io/projected/4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616-kube-api-access-2vgm9\") pod \"4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616\" (UID: \"4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616\") "
Sep 15 07:31:04.062067 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:04.061961    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616-tls-cert\") pod \"4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616\" (UID: \"4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616\") "
Sep 15 07:31:04.063925 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:04.063894    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616" (UID: "4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:31:04.063925 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:04.063902    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616-kube-api-access-2vgm9" (OuterVolumeSpecName: "kube-api-access-2vgm9") pod "4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616" (UID: "4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616"). InnerVolumeSpecName "kube-api-access-2vgm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:31:04.163420 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:04.163392    2571 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616-tls-cert\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:31:04.163420 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:04.163417    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vgm9\" (UniqueName: \"kubernetes.io/projected/4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616-kube-api-access-2vgm9\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:31:04.988937 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:04.988902    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-7v7h7"
Sep 15 07:31:04.988937 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:04.988911    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-855d77dd88-bmzpn" event={"ID":"667b93e7-d1d6-4ac0-92c4-e30040d80805","Type":"ContainerStarted","Data":"f6a24511167c3e4326fed64463d603bd0ee759cdefe5e44251f1718e4b314bbf"}
Sep 15 07:31:05.015486 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:05.015439    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-855d77dd88-bmzpn" podStartSLOduration=1.272451453 podStartE2EDuration="2.015422758s" totalImagesPullingTime="742.971305ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:31:05.014161271 +0000 UTC m=+769.560731370" watchObservedRunningTime="2026-09-15 07:31:05.015422758 +0000 UTC m=+769.561992859"
Sep 15 07:31:05.037668 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:05.037637    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-7v7h7"]
Sep 15 07:31:05.049145 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:05.049115    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-7v7h7"]
Sep 15 07:31:05.910797 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:05.910764    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616" path="/var/lib/kubelet/pods/4b8f6ff1-e6b4-4fa9-a5b4-9669f939e616/volumes"
Sep 15 07:31:05.911002 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:05.910989    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="795c2dc9-9c21-4070-8533-47e31c9c6770" path="/var/lib/kubelet/pods/795c2dc9-9c21-4070-8533-47e31c9c6770/volumes"
Sep 15 07:31:47.111095 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:47.111056    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-ghksm"]
Sep 15 07:31:47.114519 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:47.114499    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ghksm"
Sep 15 07:31:47.117598 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:47.117569    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-nk4q4\""
Sep 15 07:31:47.117725 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:47.117665    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 15 07:31:47.134581 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:47.134558    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-ghksm"]
Sep 15 07:31:47.152001 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:47.151973    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9b59cc-5710-45db-8109-8efa36ab8e3d-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-ghksm\" (UID: \"9c9b59cc-5710-45db-8109-8efa36ab8e3d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ghksm"
Sep 15 07:31:47.152134 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:47.152023    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqf96\" (UniqueName: \"kubernetes.io/projected/9c9b59cc-5710-45db-8109-8efa36ab8e3d-kube-api-access-dqf96\") pod \"payload-pre-processing-59fbc47bf8-ghksm\" (UID: \"9c9b59cc-5710-45db-8109-8efa36ab8e3d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ghksm"
Sep 15 07:31:47.252833 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:47.252793    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9b59cc-5710-45db-8109-8efa36ab8e3d-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-ghksm\" (UID: \"9c9b59cc-5710-45db-8109-8efa36ab8e3d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ghksm"
Sep 15 07:31:47.253003 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:47.252864    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqf96\" (UniqueName: \"kubernetes.io/projected/9c9b59cc-5710-45db-8109-8efa36ab8e3d-kube-api-access-dqf96\") pod \"payload-pre-processing-59fbc47bf8-ghksm\" (UID: \"9c9b59cc-5710-45db-8109-8efa36ab8e3d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ghksm"
Sep 15 07:31:47.254080 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:47.254060    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9b59cc-5710-45db-8109-8efa36ab8e3d-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-ghksm\" (UID: \"9c9b59cc-5710-45db-8109-8efa36ab8e3d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ghksm"
Sep 15 07:31:47.270008 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:47.269971    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqf96\" (UniqueName: \"kubernetes.io/projected/9c9b59cc-5710-45db-8109-8efa36ab8e3d-kube-api-access-dqf96\") pod \"payload-pre-processing-59fbc47bf8-ghksm\" (UID: \"9c9b59cc-5710-45db-8109-8efa36ab8e3d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ghksm"
Sep 15 07:31:47.425775 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:47.425673    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ghksm"
Sep 15 07:31:47.706186 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:47.706162    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-ghksm"]
Sep 15 07:31:47.708913 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:31:47.708887    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9b59cc_5710_45db_8109_8efa36ab8e3d.slice/crio-6006633d5606f7b55bd84134ac1619d164e297ed8049db9429270ad449e2391b WatchSource:0}: Error finding container 6006633d5606f7b55bd84134ac1619d164e297ed8049db9429270ad449e2391b: Status 404 returned error can't find the container with id 6006633d5606f7b55bd84134ac1619d164e297ed8049db9429270ad449e2391b
Sep 15 07:31:48.189530 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:48.189436    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ghksm" event={"ID":"9c9b59cc-5710-45db-8109-8efa36ab8e3d","Type":"ContainerStarted","Data":"6006633d5606f7b55bd84134ac1619d164e297ed8049db9429270ad449e2391b"}
Sep 15 07:31:49.196316 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:49.196285    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ghksm" event={"ID":"9c9b59cc-5710-45db-8109-8efa36ab8e3d","Type":"ContainerStarted","Data":"bce7a667c91c3c78a9fafa00efa6f447cb08e14a848c9d080c89deb72c9fe3e1"}
Sep 15 07:31:49.196741 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:49.196414    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ghksm"
Sep 15 07:31:49.212949 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:31:49.212900    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ghksm" podStartSLOduration=0.811637377 podStartE2EDuration="2.212884648s" totalImagesPullingTime="1.401247271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:31:49.211028377 +0000 UTC m=+813.757598498" watchObservedRunningTime="2026-09-15 07:31:49.212884648 +0000 UTC m=+813.759454748"
Sep 15 07:32:00.202630 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:00.202599    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ghksm"
Sep 15 07:32:18.894880 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:18.894842    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"]
Sep 15 07:32:18.899181 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:18.899066    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:18.903401 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:18.902299    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-84rbz\""
Sep 15 07:32:18.903401 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:18.902558    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 07:32:18.903401 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:18.902925    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 15 07:32:18.906206 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:18.903785    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 07:32:18.911812 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:18.911785    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"]
Sep 15 07:32:19.039795 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:19.039757    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5tbc\" (UniqueName: \"kubernetes.io/projected/a84a09c4-85e1-4440-acf7-ec9c93e9514f-kube-api-access-q5tbc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ml26w\" (UID: \"a84a09c4-85e1-4440-acf7-ec9c93e9514f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:19.039982 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:19.039815    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a84a09c4-85e1-4440-acf7-ec9c93e9514f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ml26w\" (UID: \"a84a09c4-85e1-4440-acf7-ec9c93e9514f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:19.039982 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:19.039837    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a84a09c4-85e1-4440-acf7-ec9c93e9514f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ml26w\" (UID: \"a84a09c4-85e1-4440-acf7-ec9c93e9514f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:19.039982 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:19.039855    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a84a09c4-85e1-4440-acf7-ec9c93e9514f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ml26w\" (UID: \"a84a09c4-85e1-4440-acf7-ec9c93e9514f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:19.039982 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:19.039900    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a84a09c4-85e1-4440-acf7-ec9c93e9514f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ml26w\" (UID: \"a84a09c4-85e1-4440-acf7-ec9c93e9514f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:19.039982 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:19.039978    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a84a09c4-85e1-4440-acf7-ec9c93e9514f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ml26w\" (UID: \"a84a09c4-85e1-4440-acf7-ec9c93e9514f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:19.140824 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:19.140786    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5tbc\" (UniqueName: \"kubernetes.io/projected/a84a09c4-85e1-4440-acf7-ec9c93e9514f-kube-api-access-q5tbc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ml26w\" (UID: \"a84a09c4-85e1-4440-acf7-ec9c93e9514f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:19.141019 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:19.140836    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a84a09c4-85e1-4440-acf7-ec9c93e9514f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ml26w\" (UID: \"a84a09c4-85e1-4440-acf7-ec9c93e9514f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:19.141019 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:19.140859    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a84a09c4-85e1-4440-acf7-ec9c93e9514f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ml26w\" (UID: \"a84a09c4-85e1-4440-acf7-ec9c93e9514f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:19.141019 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:19.140876    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a84a09c4-85e1-4440-acf7-ec9c93e9514f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ml26w\" (UID: \"a84a09c4-85e1-4440-acf7-ec9c93e9514f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:19.141019 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:19.140909    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a84a09c4-85e1-4440-acf7-ec9c93e9514f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ml26w\" (UID: \"a84a09c4-85e1-4440-acf7-ec9c93e9514f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:19.141019 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:19.140963    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a84a09c4-85e1-4440-acf7-ec9c93e9514f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ml26w\" (UID: \"a84a09c4-85e1-4440-acf7-ec9c93e9514f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:19.141318 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:19.141292    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a84a09c4-85e1-4440-acf7-ec9c93e9514f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ml26w\" (UID: \"a84a09c4-85e1-4440-acf7-ec9c93e9514f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:19.141401 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:19.141376    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a84a09c4-85e1-4440-acf7-ec9c93e9514f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ml26w\" (UID: \"a84a09c4-85e1-4440-acf7-ec9c93e9514f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:19.141449 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:19.141410    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a84a09c4-85e1-4440-acf7-ec9c93e9514f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ml26w\" (UID: \"a84a09c4-85e1-4440-acf7-ec9c93e9514f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:19.142970 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:19.142952    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a84a09c4-85e1-4440-acf7-ec9c93e9514f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ml26w\" (UID: \"a84a09c4-85e1-4440-acf7-ec9c93e9514f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:19.143332 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:19.143313    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a84a09c4-85e1-4440-acf7-ec9c93e9514f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ml26w\" (UID: \"a84a09c4-85e1-4440-acf7-ec9c93e9514f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:19.154148 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:19.154098    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5tbc\" (UniqueName: \"kubernetes.io/projected/a84a09c4-85e1-4440-acf7-ec9c93e9514f-kube-api-access-q5tbc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ml26w\" (UID: \"a84a09c4-85e1-4440-acf7-ec9c93e9514f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:19.216838 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:19.216808    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:19.490085 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:19.490017    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"]
Sep 15 07:32:19.492162 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:32:19.492138    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84a09c4_85e1_4440_acf7_ec9c93e9514f.slice/crio-b3654bb4288051fcd574c9bc920d8037cac348b39689ccb8804f2cafdb9b0480 WatchSource:0}: Error finding container b3654bb4288051fcd574c9bc920d8037cac348b39689ccb8804f2cafdb9b0480: Status 404 returned error can't find the container with id b3654bb4288051fcd574c9bc920d8037cac348b39689ccb8804f2cafdb9b0480
Sep 15 07:32:20.341651 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:20.341619    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w" event={"ID":"a84a09c4-85e1-4440-acf7-ec9c93e9514f","Type":"ContainerStarted","Data":"b3654bb4288051fcd574c9bc920d8037cac348b39689ccb8804f2cafdb9b0480"}
Sep 15 07:32:21.347365 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:21.347327    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w" event={"ID":"a84a09c4-85e1-4440-acf7-ec9c93e9514f","Type":"ContainerStarted","Data":"f3d7e63d038f683ab1a87c11584bba9367b97863e97617d16ef677947f5cfc8a"}
Sep 15 07:32:21.364462 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:21.364405    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w" podStartSLOduration=2.278746294 podStartE2EDuration="3.364385992s" totalImagesPullingTime="1.085639698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:32:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:32:21.363673254 +0000 UTC m=+845.910243373" watchObservedRunningTime="2026-09-15 07:32:21.364385992 +0000 UTC m=+845.910956092"
Sep 15 07:32:29.217172 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:29.217131    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:29.217588 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:29.217216    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:29.229553 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:29.229527    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:32:29.394625 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:32:29.394590    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ml26w"
Sep 15 07:33:00.600583 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.600554    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"]
Sep 15 07:33:00.604917 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.604889    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:00.607547 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.607527    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 15 07:33:00.615322 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.615239    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"]
Sep 15 07:33:00.703715 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.703680    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fcb590e-287c-4292-afc9-16fcbb5c12f8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bbcpk\" (UID: \"8fcb590e-287c-4292-afc9-16fcbb5c12f8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:00.703882 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.703727    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fcb590e-287c-4292-afc9-16fcbb5c12f8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bbcpk\" (UID: \"8fcb590e-287c-4292-afc9-16fcbb5c12f8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:00.703882 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.703785    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jlxn\" (UniqueName: \"kubernetes.io/projected/8fcb590e-287c-4292-afc9-16fcbb5c12f8-kube-api-access-4jlxn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bbcpk\" (UID: \"8fcb590e-287c-4292-afc9-16fcbb5c12f8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:00.703882 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.703844    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fcb590e-287c-4292-afc9-16fcbb5c12f8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bbcpk\" (UID: \"8fcb590e-287c-4292-afc9-16fcbb5c12f8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:00.703882 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.703875    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fcb590e-287c-4292-afc9-16fcbb5c12f8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bbcpk\" (UID: \"8fcb590e-287c-4292-afc9-16fcbb5c12f8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:00.704022 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.703914    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fcb590e-287c-4292-afc9-16fcbb5c12f8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bbcpk\" (UID: \"8fcb590e-287c-4292-afc9-16fcbb5c12f8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:00.805049 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.805005    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fcb590e-287c-4292-afc9-16fcbb5c12f8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bbcpk\" (UID: \"8fcb590e-287c-4292-afc9-16fcbb5c12f8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:00.805289 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.805066    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fcb590e-287c-4292-afc9-16fcbb5c12f8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bbcpk\" (UID: \"8fcb590e-287c-4292-afc9-16fcbb5c12f8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:00.805289 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.805097    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jlxn\" (UniqueName: \"kubernetes.io/projected/8fcb590e-287c-4292-afc9-16fcbb5c12f8-kube-api-access-4jlxn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bbcpk\" (UID: \"8fcb590e-287c-4292-afc9-16fcbb5c12f8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:00.805289 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.805136    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fcb590e-287c-4292-afc9-16fcbb5c12f8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bbcpk\" (UID: \"8fcb590e-287c-4292-afc9-16fcbb5c12f8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:00.805289 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.805165    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fcb590e-287c-4292-afc9-16fcbb5c12f8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bbcpk\" (UID: \"8fcb590e-287c-4292-afc9-16fcbb5c12f8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:00.805522 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.805304    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fcb590e-287c-4292-afc9-16fcbb5c12f8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bbcpk\" (UID: \"8fcb590e-287c-4292-afc9-16fcbb5c12f8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:00.805594 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.805573    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fcb590e-287c-4292-afc9-16fcbb5c12f8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bbcpk\" (UID: \"8fcb590e-287c-4292-afc9-16fcbb5c12f8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:00.805654 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.805638    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fcb590e-287c-4292-afc9-16fcbb5c12f8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bbcpk\" (UID: \"8fcb590e-287c-4292-afc9-16fcbb5c12f8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:00.805698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.805672    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fcb590e-287c-4292-afc9-16fcbb5c12f8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bbcpk\" (UID: \"8fcb590e-287c-4292-afc9-16fcbb5c12f8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:00.807355 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.807330    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fcb590e-287c-4292-afc9-16fcbb5c12f8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bbcpk\" (UID: \"8fcb590e-287c-4292-afc9-16fcbb5c12f8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:00.807631 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.807611    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fcb590e-287c-4292-afc9-16fcbb5c12f8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bbcpk\" (UID: \"8fcb590e-287c-4292-afc9-16fcbb5c12f8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:00.812916 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.812894    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jlxn\" (UniqueName: \"kubernetes.io/projected/8fcb590e-287c-4292-afc9-16fcbb5c12f8-kube-api-access-4jlxn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bbcpk\" (UID: \"8fcb590e-287c-4292-afc9-16fcbb5c12f8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:00.917449 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:00.917335    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:01.191346 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:01.191318    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"]
Sep 15 07:33:01.193294 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:33:01.193246    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fcb590e_287c_4292_afc9_16fcbb5c12f8.slice/crio-219c13d44820512df9a713bfb8754f04e4852ceb8f785d5d6b5f5276e949b9e8 WatchSource:0}: Error finding container 219c13d44820512df9a713bfb8754f04e4852ceb8f785d5d6b5f5276e949b9e8: Status 404 returned error can't find the container with id 219c13d44820512df9a713bfb8754f04e4852ceb8f785d5d6b5f5276e949b9e8
Sep 15 07:33:01.542609 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:01.542571    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk" event={"ID":"8fcb590e-287c-4292-afc9-16fcbb5c12f8","Type":"ContainerStarted","Data":"acab624403a6374f58ec43ce05db8c30cad170c1790e643ffa6a164d503827dd"}
Sep 15 07:33:01.542793 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:01.542668    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk" event={"ID":"8fcb590e-287c-4292-afc9-16fcbb5c12f8","Type":"ContainerStarted","Data":"219c13d44820512df9a713bfb8754f04e4852ceb8f785d5d6b5f5276e949b9e8"}
Sep 15 07:33:01.559815 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:01.559764    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk" podStartSLOduration=1.37193715 podStartE2EDuration="1.559747026s" totalImagesPullingTime="187.809876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:33:01.558307614 +0000 UTC m=+886.104877732" watchObservedRunningTime="2026-09-15 07:33:01.559747026 +0000 UTC m=+886.106317125"
Sep 15 07:33:10.917496 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:10.917411    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:10.917496 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:10.917481    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:10.929700 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:10.929668    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:11.604910 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:11.604882    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bbcpk"
Sep 15 07:33:12.909409 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:12.909368    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"]
Sep 15 07:33:12.912855 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:12.912837    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:12.915463 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:12.915442    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 15 07:33:12.920551 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:12.920528    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"]
Sep 15 07:33:12.937331 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:12.937298    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb8v5\" (UniqueName: \"kubernetes.io/projected/c04ccc49-3dfe-49a3-9435-0696d963afce-kube-api-access-kb8v5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4\" (UID: \"c04ccc49-3dfe-49a3-9435-0696d963afce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:12.937331 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:12.937330    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c04ccc49-3dfe-49a3-9435-0696d963afce-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4\" (UID: \"c04ccc49-3dfe-49a3-9435-0696d963afce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:12.937538 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:12.937411    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c04ccc49-3dfe-49a3-9435-0696d963afce-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4\" (UID: \"c04ccc49-3dfe-49a3-9435-0696d963afce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:12.937538 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:12.937460    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c04ccc49-3dfe-49a3-9435-0696d963afce-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4\" (UID: \"c04ccc49-3dfe-49a3-9435-0696d963afce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:12.937644 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:12.937535    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c04ccc49-3dfe-49a3-9435-0696d963afce-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4\" (UID: \"c04ccc49-3dfe-49a3-9435-0696d963afce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:12.937644 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:12.937597    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c04ccc49-3dfe-49a3-9435-0696d963afce-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4\" (UID: \"c04ccc49-3dfe-49a3-9435-0696d963afce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:13.038282 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:13.038212    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c04ccc49-3dfe-49a3-9435-0696d963afce-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4\" (UID: \"c04ccc49-3dfe-49a3-9435-0696d963afce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:13.038473 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:13.038310    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c04ccc49-3dfe-49a3-9435-0696d963afce-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4\" (UID: \"c04ccc49-3dfe-49a3-9435-0696d963afce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:13.038473 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:13.038357    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb8v5\" (UniqueName: \"kubernetes.io/projected/c04ccc49-3dfe-49a3-9435-0696d963afce-kube-api-access-kb8v5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4\" (UID: \"c04ccc49-3dfe-49a3-9435-0696d963afce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:13.038473 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:13.038381    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c04ccc49-3dfe-49a3-9435-0696d963afce-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4\" (UID: \"c04ccc49-3dfe-49a3-9435-0696d963afce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:13.038473 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:13.038414    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c04ccc49-3dfe-49a3-9435-0696d963afce-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4\" (UID: \"c04ccc49-3dfe-49a3-9435-0696d963afce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:13.038473 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:13.038468    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c04ccc49-3dfe-49a3-9435-0696d963afce-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4\" (UID: \"c04ccc49-3dfe-49a3-9435-0696d963afce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:13.038812 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:13.038738    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c04ccc49-3dfe-49a3-9435-0696d963afce-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4\" (UID: \"c04ccc49-3dfe-49a3-9435-0696d963afce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:13.038812 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:13.038758    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c04ccc49-3dfe-49a3-9435-0696d963afce-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4\" (UID: \"c04ccc49-3dfe-49a3-9435-0696d963afce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:13.038936 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:13.038918    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c04ccc49-3dfe-49a3-9435-0696d963afce-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4\" (UID: \"c04ccc49-3dfe-49a3-9435-0696d963afce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:13.040766 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:13.040742    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c04ccc49-3dfe-49a3-9435-0696d963afce-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4\" (UID: \"c04ccc49-3dfe-49a3-9435-0696d963afce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:13.041007 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:13.040990    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c04ccc49-3dfe-49a3-9435-0696d963afce-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4\" (UID: \"c04ccc49-3dfe-49a3-9435-0696d963afce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:13.045908 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:13.045889    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb8v5\" (UniqueName: \"kubernetes.io/projected/c04ccc49-3dfe-49a3-9435-0696d963afce-kube-api-access-kb8v5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4\" (UID: \"c04ccc49-3dfe-49a3-9435-0696d963afce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:13.225320 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:13.225275    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:13.496182 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:13.496157    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"]
Sep 15 07:33:13.498386 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:33:13.498343    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04ccc49_3dfe_49a3_9435_0696d963afce.slice/crio-7570e55d53107486ffbed52c76f9281fd92c7bcaf1808a33afa00baee8a1fc8c WatchSource:0}: Error finding container 7570e55d53107486ffbed52c76f9281fd92c7bcaf1808a33afa00baee8a1fc8c: Status 404 returned error can't find the container with id 7570e55d53107486ffbed52c76f9281fd92c7bcaf1808a33afa00baee8a1fc8c
Sep 15 07:33:13.604023 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:13.603982    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4" event={"ID":"c04ccc49-3dfe-49a3-9435-0696d963afce","Type":"ContainerStarted","Data":"7570e55d53107486ffbed52c76f9281fd92c7bcaf1808a33afa00baee8a1fc8c"}
Sep 15 07:33:14.611386 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:14.611341    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4" event={"ID":"c04ccc49-3dfe-49a3-9435-0696d963afce","Type":"ContainerStarted","Data":"c7331a9cddd5291c3840de27d9decb3a5d7444d8429b21c403cd8bf8fbf0b740"}
Sep 15 07:33:14.629200 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:14.629150    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4" podStartSLOduration=2.428783304 podStartE2EDuration="2.629132566s" totalImagesPullingTime="200.349262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:33:14.627552313 +0000 UTC m=+899.174122437" watchObservedRunningTime="2026-09-15 07:33:14.629132566 +0000 UTC m=+899.175702667"
Sep 15 07:33:23.225721 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:23.225671    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:23.226126 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:23.225747    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:23.238894 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:23.238865    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:23.672492 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:23.672415    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4"
Sep 15 07:33:50.205195 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.205155    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"]
Sep 15 07:33:50.209043 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.209017    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:33:50.211423 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.211404    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 15 07:33:50.218286 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.218245    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"]
Sep 15 07:33:50.375896 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.375856    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a80b374e-6518-4173-89a8-097cea66070f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v\" (UID: \"a80b374e-6518-4173-89a8-097cea66070f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:33:50.375896 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.375902    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a80b374e-6518-4173-89a8-097cea66070f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v\" (UID: \"a80b374e-6518-4173-89a8-097cea66070f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:33:50.376114 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.376011    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a80b374e-6518-4173-89a8-097cea66070f-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v\" (UID: \"a80b374e-6518-4173-89a8-097cea66070f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:33:50.376114 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.376047    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a80b374e-6518-4173-89a8-097cea66070f-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v\" (UID: \"a80b374e-6518-4173-89a8-097cea66070f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:33:50.376114 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.376067    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a80b374e-6518-4173-89a8-097cea66070f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v\" (UID: \"a80b374e-6518-4173-89a8-097cea66070f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:33:50.376229 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.376113    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq5gs\" (UniqueName: \"kubernetes.io/projected/a80b374e-6518-4173-89a8-097cea66070f-kube-api-access-xq5gs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v\" (UID: \"a80b374e-6518-4173-89a8-097cea66070f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:33:50.477433 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.477347    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a80b374e-6518-4173-89a8-097cea66070f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v\" (UID: \"a80b374e-6518-4173-89a8-097cea66070f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:33:50.477433 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.477402    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a80b374e-6518-4173-89a8-097cea66070f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v\" (UID: \"a80b374e-6518-4173-89a8-097cea66070f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:33:50.477640 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.477479    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a80b374e-6518-4173-89a8-097cea66070f-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v\" (UID: \"a80b374e-6518-4173-89a8-097cea66070f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:33:50.477640 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.477508    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a80b374e-6518-4173-89a8-097cea66070f-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v\" (UID: \"a80b374e-6518-4173-89a8-097cea66070f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:33:50.477640 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.477536    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a80b374e-6518-4173-89a8-097cea66070f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v\" (UID: \"a80b374e-6518-4173-89a8-097cea66070f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:33:50.477640 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.477620    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq5gs\" (UniqueName: \"kubernetes.io/projected/a80b374e-6518-4173-89a8-097cea66070f-kube-api-access-xq5gs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v\" (UID: \"a80b374e-6518-4173-89a8-097cea66070f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:33:50.477916 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.477890    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a80b374e-6518-4173-89a8-097cea66070f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v\" (UID: \"a80b374e-6518-4173-89a8-097cea66070f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:33:50.477916 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.477908    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a80b374e-6518-4173-89a8-097cea66070f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v\" (UID: \"a80b374e-6518-4173-89a8-097cea66070f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:33:50.478051 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.477944    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a80b374e-6518-4173-89a8-097cea66070f-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v\" (UID: \"a80b374e-6518-4173-89a8-097cea66070f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:33:50.479709 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.479685    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a80b374e-6518-4173-89a8-097cea66070f-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v\" (UID: \"a80b374e-6518-4173-89a8-097cea66070f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:33:50.479966 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.479947    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a80b374e-6518-4173-89a8-097cea66070f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v\" (UID: \"a80b374e-6518-4173-89a8-097cea66070f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:33:50.487972 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.487948    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq5gs\" (UniqueName: \"kubernetes.io/projected/a80b374e-6518-4173-89a8-097cea66070f-kube-api-access-xq5gs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v\" (UID: \"a80b374e-6518-4173-89a8-097cea66070f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:33:50.518774 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.518738    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:33:50.787065 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.787041    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"]
Sep 15 07:33:50.789294 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:33:50.789244    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda80b374e_6518_4173_89a8_097cea66070f.slice/crio-7b078fec618ee398460e501f8183a344da8e04fbfa8ad45f2c11f66da1576360 WatchSource:0}: Error finding container 7b078fec618ee398460e501f8183a344da8e04fbfa8ad45f2c11f66da1576360: Status 404 returned error can't find the container with id 7b078fec618ee398460e501f8183a344da8e04fbfa8ad45f2c11f66da1576360
Sep 15 07:33:50.798180 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:50.798154    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v" event={"ID":"a80b374e-6518-4173-89a8-097cea66070f","Type":"ContainerStarted","Data":"7b078fec618ee398460e501f8183a344da8e04fbfa8ad45f2c11f66da1576360"}
Sep 15 07:33:51.810453 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:51.810417    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v" event={"ID":"a80b374e-6518-4173-89a8-097cea66070f","Type":"ContainerStarted","Data":"16e944e13cae2da85adc8f57aa53b524b4b35ee19cc02181bed832e38d0e144a"}
Sep 15 07:33:51.829360 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:33:51.829312    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v" podStartSLOduration=1.663254349 podStartE2EDuration="1.829297463s" totalImagesPullingTime="166.043114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:33:51.827057266 +0000 UTC m=+936.373627377" watchObservedRunningTime="2026-09-15 07:33:51.829297463 +0000 UTC m=+936.375867604"
Sep 15 07:34:00.519805 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:00.519768    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:34:00.520189 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:00.519819    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:34:00.532378 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:00.532351    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:34:00.865422 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:00.865336    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v"
Sep 15 07:34:56.562382 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:56.562347    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-855d77dd88-bmzpn"]
Sep 15 07:34:56.562843 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:56.562580    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-855d77dd88-bmzpn" podUID="667b93e7-d1d6-4ac0-92c4-e30040d80805" containerName="authorino" containerID="cri-o://f6a24511167c3e4326fed64463d603bd0ee759cdefe5e44251f1718e4b314bbf" gracePeriod=30
Sep 15 07:34:56.805066 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:56.805042    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-855d77dd88-bmzpn"
Sep 15 07:34:56.870962 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:56.870880    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/667b93e7-d1d6-4ac0-92c4-e30040d80805-tls-cert\") pod \"667b93e7-d1d6-4ac0-92c4-e30040d80805\" (UID: \"667b93e7-d1d6-4ac0-92c4-e30040d80805\") "
Sep 15 07:34:56.870962 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:56.870948    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mphb\" (UniqueName: \"kubernetes.io/projected/667b93e7-d1d6-4ac0-92c4-e30040d80805-kube-api-access-2mphb\") pod \"667b93e7-d1d6-4ac0-92c4-e30040d80805\" (UID: \"667b93e7-d1d6-4ac0-92c4-e30040d80805\") "
Sep 15 07:34:56.872904 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:56.872876    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667b93e7-d1d6-4ac0-92c4-e30040d80805-kube-api-access-2mphb" (OuterVolumeSpecName: "kube-api-access-2mphb") pod "667b93e7-d1d6-4ac0-92c4-e30040d80805" (UID: "667b93e7-d1d6-4ac0-92c4-e30040d80805"). InnerVolumeSpecName "kube-api-access-2mphb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:34:56.880997 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:56.880974    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/667b93e7-d1d6-4ac0-92c4-e30040d80805-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "667b93e7-d1d6-4ac0-92c4-e30040d80805" (UID: "667b93e7-d1d6-4ac0-92c4-e30040d80805"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:34:56.971537 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:56.971502    2571 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/667b93e7-d1d6-4ac0-92c4-e30040d80805-tls-cert\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:34:56.971537 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:56.971533    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mphb\" (UniqueName: \"kubernetes.io/projected/667b93e7-d1d6-4ac0-92c4-e30040d80805-kube-api-access-2mphb\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:34:57.135485 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:57.135391    2571 generic.go:356] "Generic (PLEG): container finished" podID="667b93e7-d1d6-4ac0-92c4-e30040d80805" containerID="f6a24511167c3e4326fed64463d603bd0ee759cdefe5e44251f1718e4b314bbf" exitCode=0
Sep 15 07:34:57.135485 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:57.135435    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-855d77dd88-bmzpn"
Sep 15 07:34:57.135724 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:57.135482    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-855d77dd88-bmzpn" event={"ID":"667b93e7-d1d6-4ac0-92c4-e30040d80805","Type":"ContainerDied","Data":"f6a24511167c3e4326fed64463d603bd0ee759cdefe5e44251f1718e4b314bbf"}
Sep 15 07:34:57.135724 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:57.135533    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-855d77dd88-bmzpn" event={"ID":"667b93e7-d1d6-4ac0-92c4-e30040d80805","Type":"ContainerDied","Data":"fc0489eddd9bfc2b2b1bf1bfd030b73011f72381ce29ed7c659f919e78238aa4"}
Sep 15 07:34:57.135724 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:57.135555    2571 scope.go:117] "RemoveContainer" containerID="f6a24511167c3e4326fed64463d603bd0ee759cdefe5e44251f1718e4b314bbf"
Sep 15 07:34:57.146566 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:57.146534    2571 scope.go:117] "RemoveContainer" containerID="f6a24511167c3e4326fed64463d603bd0ee759cdefe5e44251f1718e4b314bbf"
Sep 15 07:34:57.147041 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:34:57.147020    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6a24511167c3e4326fed64463d603bd0ee759cdefe5e44251f1718e4b314bbf\": container with ID starting with f6a24511167c3e4326fed64463d603bd0ee759cdefe5e44251f1718e4b314bbf not found: ID does not exist" containerID="f6a24511167c3e4326fed64463d603bd0ee759cdefe5e44251f1718e4b314bbf"
Sep 15 07:34:57.147118 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:57.147047    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a24511167c3e4326fed64463d603bd0ee759cdefe5e44251f1718e4b314bbf"} err="failed to get container status \"f6a24511167c3e4326fed64463d603bd0ee759cdefe5e44251f1718e4b314bbf\": rpc error: code = NotFound desc = could not find container \"f6a24511167c3e4326fed64463d603bd0ee759cdefe5e44251f1718e4b314bbf\": container with ID starting with f6a24511167c3e4326fed64463d603bd0ee759cdefe5e44251f1718e4b314bbf not found: ID does not exist"
Sep 15 07:34:57.162413 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:57.162388    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-855d77dd88-bmzpn"]
Sep 15 07:34:57.165946 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:57.165920    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-855d77dd88-bmzpn"]
Sep 15 07:34:57.909857 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:34:57.909811    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="667b93e7-d1d6-4ac0-92c4-e30040d80805" path="/var/lib/kubelet/pods/667b93e7-d1d6-4ac0-92c4-e30040d80805/volumes"
Sep 15 07:35:19.701010 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:19.700974    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww"]
Sep 15 07:35:19.701516 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:19.701438    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="667b93e7-d1d6-4ac0-92c4-e30040d80805" containerName="authorino"
Sep 15 07:35:19.701516 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:19.701457    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="667b93e7-d1d6-4ac0-92c4-e30040d80805" containerName="authorino"
Sep 15 07:35:19.701637 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:19.701543    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="667b93e7-d1d6-4ac0-92c4-e30040d80805" containerName="authorino"
Sep 15 07:35:19.705955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:19.705932    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww"
Sep 15 07:35:19.708522 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:19.708503    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-2735c5-dockercfg-4lfvh\""
Sep 15 07:35:19.708522 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:19.708513    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-2735c5\""
Sep 15 07:35:19.716448 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:19.716421    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww"]
Sep 15 07:35:19.773607 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:19.773576    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5mfv\" (UniqueName: \"kubernetes.io/projected/d6c1f080-389f-4ab8-abd7-e62804e7302f-kube-api-access-k5mfv\") pod \"payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww\" (UID: \"d6c1f080-389f-4ab8-abd7-e62804e7302f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww"
Sep 15 07:35:19.773871 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:19.773623    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c1f080-389f-4ab8-abd7-e62804e7302f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww\" (UID: \"d6c1f080-389f-4ab8-abd7-e62804e7302f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww"
Sep 15 07:35:19.874491 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:19.874453    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5mfv\" (UniqueName: \"kubernetes.io/projected/d6c1f080-389f-4ab8-abd7-e62804e7302f-kube-api-access-k5mfv\") pod \"payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww\" (UID: \"d6c1f080-389f-4ab8-abd7-e62804e7302f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww"
Sep 15 07:35:19.874661 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:19.874505    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c1f080-389f-4ab8-abd7-e62804e7302f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww\" (UID: \"d6c1f080-389f-4ab8-abd7-e62804e7302f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww"
Sep 15 07:35:19.875133 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:19.875114    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c1f080-389f-4ab8-abd7-e62804e7302f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww\" (UID: \"d6c1f080-389f-4ab8-abd7-e62804e7302f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww"
Sep 15 07:35:19.882129 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:19.882101    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5mfv\" (UniqueName: \"kubernetes.io/projected/d6c1f080-389f-4ab8-abd7-e62804e7302f-kube-api-access-k5mfv\") pod \"payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww\" (UID: \"d6c1f080-389f-4ab8-abd7-e62804e7302f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww"
Sep 15 07:35:20.018229 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:20.018143    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww"
Sep 15 07:35:20.289725 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:20.289698    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww"]
Sep 15 07:35:20.292333 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:35:20.292300    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c1f080_389f_4ab8_abd7_e62804e7302f.slice/crio-69475d03b6e9a85f71b952312d5d88e887fc3b9321b5929b79b0c83b3d26a9f4 WatchSource:0}: Error finding container 69475d03b6e9a85f71b952312d5d88e887fc3b9321b5929b79b0c83b3d26a9f4: Status 404 returned error can't find the container with id 69475d03b6e9a85f71b952312d5d88e887fc3b9321b5929b79b0c83b3d26a9f4
Sep 15 07:35:20.753797 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:20.753758    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr"]
Sep 15 07:35:20.757422 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:20.757403    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr"
Sep 15 07:35:20.759825 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:20.759798    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-f56165-dockercfg-ltc5x\""
Sep 15 07:35:20.759825 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:20.759822    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-f56165\""
Sep 15 07:35:20.770397 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:20.770374    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr"]
Sep 15 07:35:20.782004 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:20.781965    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kc9v\" (UniqueName: \"kubernetes.io/projected/3e704c76-3c4d-4e0d-83b4-c0e9115f33c1-kube-api-access-4kc9v\") pod \"payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr\" (UID: \"3e704c76-3c4d-4e0d-83b4-c0e9115f33c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr"
Sep 15 07:35:20.782140 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:20.782096    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e704c76-3c4d-4e0d-83b4-c0e9115f33c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr\" (UID: \"3e704c76-3c4d-4e0d-83b4-c0e9115f33c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr"
Sep 15 07:35:20.783245 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:20.783222    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26"]
Sep 15 07:35:20.787178 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:20.787154    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26"
Sep 15 07:35:20.800348 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:20.800320    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26"]
Sep 15 07:35:20.882576 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:20.882536    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/827d14fe-b570-49ca-8600-778f1146b956-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26\" (UID: \"827d14fe-b570-49ca-8600-778f1146b956\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26"
Sep 15 07:35:20.882734 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:20.882587    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kc9v\" (UniqueName: \"kubernetes.io/projected/3e704c76-3c4d-4e0d-83b4-c0e9115f33c1-kube-api-access-4kc9v\") pod \"payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr\" (UID: \"3e704c76-3c4d-4e0d-83b4-c0e9115f33c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr"
Sep 15 07:35:20.882734 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:20.882622    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5ms5\" (UniqueName: \"kubernetes.io/projected/827d14fe-b570-49ca-8600-778f1146b956-kube-api-access-p5ms5\") pod \"payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26\" (UID: \"827d14fe-b570-49ca-8600-778f1146b956\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26"
Sep 15 07:35:20.882734 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:20.882700    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e704c76-3c4d-4e0d-83b4-c0e9115f33c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr\" (UID: \"3e704c76-3c4d-4e0d-83b4-c0e9115f33c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr"
Sep 15 07:35:20.883400 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:20.883380    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e704c76-3c4d-4e0d-83b4-c0e9115f33c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr\" (UID: \"3e704c76-3c4d-4e0d-83b4-c0e9115f33c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr"
Sep 15 07:35:20.890409 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:20.890385    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kc9v\" (UniqueName: \"kubernetes.io/projected/3e704c76-3c4d-4e0d-83b4-c0e9115f33c1-kube-api-access-4kc9v\") pod \"payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr\" (UID: \"3e704c76-3c4d-4e0d-83b4-c0e9115f33c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr"
Sep 15 07:35:20.983724 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:20.983686    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/827d14fe-b570-49ca-8600-778f1146b956-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26\" (UID: \"827d14fe-b570-49ca-8600-778f1146b956\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26"
Sep 15 07:35:20.983888 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:20.983738    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5ms5\" (UniqueName: \"kubernetes.io/projected/827d14fe-b570-49ca-8600-778f1146b956-kube-api-access-p5ms5\") pod \"payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26\" (UID: \"827d14fe-b570-49ca-8600-778f1146b956\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26"
Sep 15 07:35:20.984396 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:20.984376    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/827d14fe-b570-49ca-8600-778f1146b956-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26\" (UID: \"827d14fe-b570-49ca-8600-778f1146b956\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26"
Sep 15 07:35:20.991749 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:20.991726    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5ms5\" (UniqueName: \"kubernetes.io/projected/827d14fe-b570-49ca-8600-778f1146b956-kube-api-access-p5ms5\") pod \"payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26\" (UID: \"827d14fe-b570-49ca-8600-778f1146b956\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26"
Sep 15 07:35:21.068369 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:21.068288    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr"
Sep 15 07:35:21.101814 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:21.101775    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26"
Sep 15 07:35:21.254011 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:21.253963    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww" event={"ID":"d6c1f080-389f-4ab8-abd7-e62804e7302f","Type":"ContainerStarted","Data":"f74c951805a2495a829abe383ce6a186771e10e4d759c365d9511c92df071e0b"}
Sep 15 07:35:21.254182 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:21.254033    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww" event={"ID":"d6c1f080-389f-4ab8-abd7-e62804e7302f","Type":"ContainerStarted","Data":"69475d03b6e9a85f71b952312d5d88e887fc3b9321b5929b79b0c83b3d26a9f4"}
Sep 15 07:35:21.254226 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:21.254180    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww"
Sep 15 07:35:21.273588 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:21.273541    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww" podStartSLOduration=2.273527769 podStartE2EDuration="2.273527769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:35:21.272416155 +0000 UTC m=+1025.818986274" watchObservedRunningTime="2026-09-15 07:35:21.273527769 +0000 UTC m=+1025.820097865"
Sep 15 07:35:21.346291 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:21.343983    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr"]
Sep 15 07:35:21.383825 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:21.383792    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26"]
Sep 15 07:35:21.385282 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:35:21.385244    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827d14fe_b570_49ca_8600_778f1146b956.slice/crio-ae063872b5a4541bd5d45d5c9ea9366ec1a773d1981237fba7e9748b3d43db95 WatchSource:0}: Error finding container ae063872b5a4541bd5d45d5c9ea9366ec1a773d1981237fba7e9748b3d43db95: Status 404 returned error can't find the container with id ae063872b5a4541bd5d45d5c9ea9366ec1a773d1981237fba7e9748b3d43db95
Sep 15 07:35:21.784526 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:21.784491    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg"]
Sep 15 07:35:21.788671 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:21.788648    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg"
Sep 15 07:35:21.791515 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:21.791488    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ee0096\""
Sep 15 07:35:21.791647 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:21.791497    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ee0096-dockercfg-tsd4g\""
Sep 15 07:35:21.800158 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:21.800128    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg"]
Sep 15 07:35:21.892296 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:21.892210    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glmzq\" (UniqueName: \"kubernetes.io/projected/c6d1224c-ff36-4220-ab57-63666a719501-kube-api-access-glmzq\") pod \"payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg\" (UID: \"c6d1224c-ff36-4220-ab57-63666a719501\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg"
Sep 15 07:35:21.892489 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:21.892322    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d1224c-ff36-4220-ab57-63666a719501-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg\" (UID: \"c6d1224c-ff36-4220-ab57-63666a719501\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg"
Sep 15 07:35:21.993053 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:21.993019    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glmzq\" (UniqueName: \"kubernetes.io/projected/c6d1224c-ff36-4220-ab57-63666a719501-kube-api-access-glmzq\") pod \"payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg\" (UID: \"c6d1224c-ff36-4220-ab57-63666a719501\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg"
Sep 15 07:35:21.993206 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:21.993078    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d1224c-ff36-4220-ab57-63666a719501-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg\" (UID: \"c6d1224c-ff36-4220-ab57-63666a719501\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg"
Sep 15 07:35:21.993800 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:21.993779    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d1224c-ff36-4220-ab57-63666a719501-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg\" (UID: \"c6d1224c-ff36-4220-ab57-63666a719501\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg"
Sep 15 07:35:22.001151 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:22.001122    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glmzq\" (UniqueName: \"kubernetes.io/projected/c6d1224c-ff36-4220-ab57-63666a719501-kube-api-access-glmzq\") pod \"payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg\" (UID: \"c6d1224c-ff36-4220-ab57-63666a719501\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg"
Sep 15 07:35:22.105937 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:22.105856    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg"
Sep 15 07:35:22.263786 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:22.263738    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr" event={"ID":"3e704c76-3c4d-4e0d-83b4-c0e9115f33c1","Type":"ContainerStarted","Data":"ccbc8d073f845e3dfd86da331719a0964903fbee4338bf95c37df81df291a1de"}
Sep 15 07:35:22.263955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:22.263808    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr"
Sep 15 07:35:22.263955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:22.263834    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr" event={"ID":"3e704c76-3c4d-4e0d-83b4-c0e9115f33c1","Type":"ContainerStarted","Data":"c11ba94f96dd6918023db6fa5c99a5524f672556414d4128b17d79ec1627b248"}
Sep 15 07:35:22.265227 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:22.265191    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26" event={"ID":"827d14fe-b570-49ca-8600-778f1146b956","Type":"ContainerStarted","Data":"74ddd16112e4aa739bcdde7cf119e269ac64f2ba669a2f9489fb48efc5ee31b6"}
Sep 15 07:35:22.265362 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:22.265242    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26" event={"ID":"827d14fe-b570-49ca-8600-778f1146b956","Type":"ContainerStarted","Data":"ae063872b5a4541bd5d45d5c9ea9366ec1a773d1981237fba7e9748b3d43db95"}
Sep 15 07:35:22.265410 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:22.265371    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26"
Sep 15 07:35:22.280898 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:22.280852    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr" podStartSLOduration=2.280836648 podStartE2EDuration="2.280836648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:35:22.279811723 +0000 UTC m=+1026.826381836" watchObservedRunningTime="2026-09-15 07:35:22.280836648 +0000 UTC m=+1026.827406747"
Sep 15 07:35:22.299028 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:22.298981    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26" podStartSLOduration=2.298967725 podStartE2EDuration="2.298967725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:35:22.295839932 +0000 UTC m=+1026.842410054" watchObservedRunningTime="2026-09-15 07:35:22.298967725 +0000 UTC m=+1026.845537824"
Sep 15 07:35:22.380103 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:22.379774    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg"]
Sep 15 07:35:22.382773 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:35:22.382607    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d1224c_ff36_4220_ab57_63666a719501.slice/crio-f9f4518237e0ff814370cc7cf16db4337e5caaf653d3e38632b5120c0dceaf38 WatchSource:0}: Error finding container f9f4518237e0ff814370cc7cf16db4337e5caaf653d3e38632b5120c0dceaf38: Status 404 returned error can't find the container with id f9f4518237e0ff814370cc7cf16db4337e5caaf653d3e38632b5120c0dceaf38
Sep 15 07:35:23.273200 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:23.273158    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg" event={"ID":"c6d1224c-ff36-4220-ab57-63666a719501","Type":"ContainerStarted","Data":"a95c227bb1f6dad9b36947ea34d02a5ccfa26206b6b55ac2fbf21fc0fe85c760"}
Sep 15 07:35:23.273642 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:23.273539    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg"
Sep 15 07:35:23.273642 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:23.273579    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg" event={"ID":"c6d1224c-ff36-4220-ab57-63666a719501","Type":"ContainerStarted","Data":"f9f4518237e0ff814370cc7cf16db4337e5caaf653d3e38632b5120c0dceaf38"}
Sep 15 07:35:23.289513 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:23.289461    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg" podStartSLOduration=2.289444543 podStartE2EDuration="2.289444543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:35:23.288761434 +0000 UTC m=+1027.835331561" watchObservedRunningTime="2026-09-15 07:35:23.289444543 +0000 UTC m=+1027.836014643"
Sep 15 07:35:29.655933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:29.655895    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p"]
Sep 15 07:35:29.659437 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:29.659419    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p"
Sep 15 07:35:29.667275 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:29.662058    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e8faa2dc-dockercfg-bpc6s\""
Sep 15 07:35:29.667275 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:29.662428    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e8faa2dc\""
Sep 15 07:35:29.671073 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:29.670962    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p"]
Sep 15 07:35:29.737646 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:29.737609    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j"]
Sep 15 07:35:29.741310 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:29.741289    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j"
Sep 15 07:35:29.753442 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:29.753420    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j"]
Sep 15 07:35:29.770645 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:29.770619    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k7r5\" (UniqueName: \"kubernetes.io/projected/70c55d24-de21-4add-bdc3-1dfd87c0bd9c-kube-api-access-2k7r5\") pod \"payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p\" (UID: \"70c55d24-de21-4add-bdc3-1dfd87c0bd9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p"
Sep 15 07:35:29.770773 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:29.770685    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70c55d24-de21-4add-bdc3-1dfd87c0bd9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p\" (UID: \"70c55d24-de21-4add-bdc3-1dfd87c0bd9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p"
Sep 15 07:35:29.871643 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:29.871609    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70c55d24-de21-4add-bdc3-1dfd87c0bd9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p\" (UID: \"70c55d24-de21-4add-bdc3-1dfd87c0bd9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p"
Sep 15 07:35:29.871643 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:29.871645    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9548c820-e25e-4a05-84ca-627208e624f0-plugins-config-volume\") pod \"payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j\" (UID: \"9548c820-e25e-4a05-84ca-627208e624f0\") " pod="openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j"
Sep 15 07:35:29.871846 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:29.871692    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj7l7\" (UniqueName: \"kubernetes.io/projected/9548c820-e25e-4a05-84ca-627208e624f0-kube-api-access-jj7l7\") pod \"payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j\" (UID: \"9548c820-e25e-4a05-84ca-627208e624f0\") " pod="openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j"
Sep 15 07:35:29.871846 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:29.871723    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k7r5\" (UniqueName: \"kubernetes.io/projected/70c55d24-de21-4add-bdc3-1dfd87c0bd9c-kube-api-access-2k7r5\") pod \"payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p\" (UID: \"70c55d24-de21-4add-bdc3-1dfd87c0bd9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p"
Sep 15 07:35:29.872435 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:29.872415    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70c55d24-de21-4add-bdc3-1dfd87c0bd9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p\" (UID: \"70c55d24-de21-4add-bdc3-1dfd87c0bd9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p"
Sep 15 07:35:29.879574 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:29.879548    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k7r5\" (UniqueName: \"kubernetes.io/projected/70c55d24-de21-4add-bdc3-1dfd87c0bd9c-kube-api-access-2k7r5\") pod \"payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p\" (UID: \"70c55d24-de21-4add-bdc3-1dfd87c0bd9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p"
Sep 15 07:35:29.973318 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:29.973281    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj7l7\" (UniqueName: \"kubernetes.io/projected/9548c820-e25e-4a05-84ca-627208e624f0-kube-api-access-jj7l7\") pod \"payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j\" (UID: \"9548c820-e25e-4a05-84ca-627208e624f0\") " pod="openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j"
Sep 15 07:35:29.973497 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:29.973388    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9548c820-e25e-4a05-84ca-627208e624f0-plugins-config-volume\") pod \"payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j\" (UID: \"9548c820-e25e-4a05-84ca-627208e624f0\") " pod="openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j"
Sep 15 07:35:29.974094 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:29.974074    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9548c820-e25e-4a05-84ca-627208e624f0-plugins-config-volume\") pod \"payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j\" (UID: \"9548c820-e25e-4a05-84ca-627208e624f0\") " pod="openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j"
Sep 15 07:35:29.978128 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:29.978103    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p"
Sep 15 07:35:29.980985 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:29.980961    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj7l7\" (UniqueName: \"kubernetes.io/projected/9548c820-e25e-4a05-84ca-627208e624f0-kube-api-access-jj7l7\") pod \"payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j\" (UID: \"9548c820-e25e-4a05-84ca-627208e624f0\") " pod="openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j"
Sep 15 07:35:30.051875 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:30.051845    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j"
Sep 15 07:35:30.252844 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:30.252820    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p"]
Sep 15 07:35:30.255444 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:35:30.255420    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c55d24_de21_4add_bdc3_1dfd87c0bd9c.slice/crio-9938c543c7e2469d577c4ccc1db6ab9c001d667573cde23bdc55ef1985ed5b07 WatchSource:0}: Error finding container 9938c543c7e2469d577c4ccc1db6ab9c001d667573cde23bdc55ef1985ed5b07: Status 404 returned error can't find the container with id 9938c543c7e2469d577c4ccc1db6ab9c001d667573cde23bdc55ef1985ed5b07
Sep 15 07:35:30.312524 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:30.312485    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p" event={"ID":"70c55d24-de21-4add-bdc3-1dfd87c0bd9c","Type":"ContainerStarted","Data":"9938c543c7e2469d577c4ccc1db6ab9c001d667573cde23bdc55ef1985ed5b07"}
Sep 15 07:35:30.333701 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:30.333669    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j"]
Sep 15 07:35:30.335913 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:35:30.335886    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9548c820_e25e_4a05_84ca_627208e624f0.slice/crio-562e4df6b04a05464861d78d09fb2c59ba0019f55d85e51da59a96c7d5acf84d WatchSource:0}: Error finding container 562e4df6b04a05464861d78d09fb2c59ba0019f55d85e51da59a96c7d5acf84d: Status 404 returned error can't find the container with id 562e4df6b04a05464861d78d09fb2c59ba0019f55d85e51da59a96c7d5acf84d
Sep 15 07:35:31.321316 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:31.321246    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p" event={"ID":"70c55d24-de21-4add-bdc3-1dfd87c0bd9c","Type":"ContainerStarted","Data":"b09a66b20ead14f63fb1f07b74e295f0c5228a5e637cfd72ba4319c460893773"}
Sep 15 07:35:31.321807 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:31.321359    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p"
Sep 15 07:35:31.323109 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:31.323084    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j" event={"ID":"9548c820-e25e-4a05-84ca-627208e624f0","Type":"ContainerStarted","Data":"bbc99aef91686b0a0172105082a26a9ef49d85073b6473abf7e52363bda3ec65"}
Sep 15 07:35:31.323249 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:31.323124    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j" event={"ID":"9548c820-e25e-4a05-84ca-627208e624f0","Type":"ContainerStarted","Data":"562e4df6b04a05464861d78d09fb2c59ba0019f55d85e51da59a96c7d5acf84d"}
Sep 15 07:35:31.323249 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:31.323157    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j"
Sep 15 07:35:31.340194 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:31.340141    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p" podStartSLOduration=2.340123135 podStartE2EDuration="2.340123135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:35:31.338089266 +0000 UTC m=+1035.884659385" watchObservedRunningTime="2026-09-15 07:35:31.340123135 +0000 UTC m=+1035.886693239"
Sep 15 07:35:31.356310 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:31.356242    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j" podStartSLOduration=2.356224869 podStartE2EDuration="2.356224869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:35:31.353864804 +0000 UTC m=+1035.900434919" watchObservedRunningTime="2026-09-15 07:35:31.356224869 +0000 UTC m=+1035.902794969"
Sep 15 07:35:32.059025 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:32.058983    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx"]
Sep 15 07:35:32.062859 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:32.062843    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx"
Sep 15 07:35:32.065211 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:32.065189    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-f33ec8-dockercfg-b9576\""
Sep 15 07:35:32.065296 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:32.065221    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-f33ec8\""
Sep 15 07:35:32.074003 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:32.073978    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx"]
Sep 15 07:35:32.195596 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:32.195561    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a72daa9a-2303-445d-9aad-717a590380e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx\" (UID: \"a72daa9a-2303-445d-9aad-717a590380e9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx"
Sep 15 07:35:32.195764 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:32.195610    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxh85\" (UniqueName: \"kubernetes.io/projected/a72daa9a-2303-445d-9aad-717a590380e9-kube-api-access-hxh85\") pod \"payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx\" (UID: \"a72daa9a-2303-445d-9aad-717a590380e9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx"
Sep 15 07:35:32.267452 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:32.267423    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww"
Sep 15 07:35:32.296621 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:32.296590    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a72daa9a-2303-445d-9aad-717a590380e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx\" (UID: \"a72daa9a-2303-445d-9aad-717a590380e9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx"
Sep 15 07:35:32.296621 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:32.296629    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxh85\" (UniqueName: \"kubernetes.io/projected/a72daa9a-2303-445d-9aad-717a590380e9-kube-api-access-hxh85\") pod \"payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx\" (UID: \"a72daa9a-2303-445d-9aad-717a590380e9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx"
Sep 15 07:35:32.297310 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:32.297288    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a72daa9a-2303-445d-9aad-717a590380e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx\" (UID: \"a72daa9a-2303-445d-9aad-717a590380e9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx"
Sep 15 07:35:32.305077 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:32.305051    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxh85\" (UniqueName: \"kubernetes.io/projected/a72daa9a-2303-445d-9aad-717a590380e9-kube-api-access-hxh85\") pod \"payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx\" (UID: \"a72daa9a-2303-445d-9aad-717a590380e9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx"
Sep 15 07:35:32.374095 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:32.374015    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx"
Sep 15 07:35:32.645060 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:32.645033    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx"]
Sep 15 07:35:32.646876 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:35:32.646852    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72daa9a_2303_445d_9aad_717a590380e9.slice/crio-410e05ec278ecc1cce7cad36843b0fa1ab05cea2ced2c88cbb875a0dbd237e17 WatchSource:0}: Error finding container 410e05ec278ecc1cce7cad36843b0fa1ab05cea2ced2c88cbb875a0dbd237e17: Status 404 returned error can't find the container with id 410e05ec278ecc1cce7cad36843b0fa1ab05cea2ced2c88cbb875a0dbd237e17
Sep 15 07:35:33.275672 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:33.275638    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr"
Sep 15 07:35:33.275864 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:33.275718    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26"
Sep 15 07:35:33.336264 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:33.336212    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx" event={"ID":"a72daa9a-2303-445d-9aad-717a590380e9","Type":"ContainerStarted","Data":"f1844b0b8c899b50c110d2d8d3ebb48ba9eb5b09631aa91091d32627fe1d8df3"}
Sep 15 07:35:33.336428 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:33.336340    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx"
Sep 15 07:35:33.336428 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:33.336373    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx" event={"ID":"a72daa9a-2303-445d-9aad-717a590380e9","Type":"ContainerStarted","Data":"410e05ec278ecc1cce7cad36843b0fa1ab05cea2ced2c88cbb875a0dbd237e17"}
Sep 15 07:35:33.355788 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:33.355731    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx" podStartSLOduration=1.355715185 podStartE2EDuration="1.355715185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:35:33.353703784 +0000 UTC m=+1037.900273883" watchObservedRunningTime="2026-09-15 07:35:33.355715185 +0000 UTC m=+1037.902285284"
Sep 15 07:35:35.286028 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:35.285995    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg"
Sep 15 07:35:39.511645 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:39.511611    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx"]
Sep 15 07:35:39.515485 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:39.515468    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx"
Sep 15 07:35:39.519390 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:39.519370    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-fcd453\""
Sep 15 07:35:39.519749 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:39.519728    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-fcd453-dockercfg-phkk9\""
Sep 15 07:35:39.528836 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:39.528810    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx"]
Sep 15 07:35:39.668120 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:39.668083    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95397f4f-11d5-4393-94ce-3d29ea930c5a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx\" (UID: \"95397f4f-11d5-4393-94ce-3d29ea930c5a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx"
Sep 15 07:35:39.668326 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:39.668164    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvnvp\" (UniqueName: \"kubernetes.io/projected/95397f4f-11d5-4393-94ce-3d29ea930c5a-kube-api-access-bvnvp\") pod \"payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx\" (UID: \"95397f4f-11d5-4393-94ce-3d29ea930c5a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx"
Sep 15 07:35:39.768761 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:39.768679    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvnvp\" (UniqueName: \"kubernetes.io/projected/95397f4f-11d5-4393-94ce-3d29ea930c5a-kube-api-access-bvnvp\") pod \"payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx\" (UID: \"95397f4f-11d5-4393-94ce-3d29ea930c5a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx"
Sep 15 07:35:39.768761 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:39.768738    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95397f4f-11d5-4393-94ce-3d29ea930c5a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx\" (UID: \"95397f4f-11d5-4393-94ce-3d29ea930c5a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx"
Sep 15 07:35:39.769408 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:39.769389    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95397f4f-11d5-4393-94ce-3d29ea930c5a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx\" (UID: \"95397f4f-11d5-4393-94ce-3d29ea930c5a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx"
Sep 15 07:35:39.778105 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:39.778072    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvnvp\" (UniqueName: \"kubernetes.io/projected/95397f4f-11d5-4393-94ce-3d29ea930c5a-kube-api-access-bvnvp\") pod \"payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx\" (UID: \"95397f4f-11d5-4393-94ce-3d29ea930c5a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx"
Sep 15 07:35:39.826066 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:39.826020    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx"
Sep 15 07:35:40.100226 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:40.100201    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx"]
Sep 15 07:35:40.102309 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:35:40.102245    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95397f4f_11d5_4393_94ce_3d29ea930c5a.slice/crio-ee90f670116b45a57148a0574a7a3dafa9bf57d64aefea801187b560122ef2e8 WatchSource:0}: Error finding container ee90f670116b45a57148a0574a7a3dafa9bf57d64aefea801187b560122ef2e8: Status 404 returned error can't find the container with id ee90f670116b45a57148a0574a7a3dafa9bf57d64aefea801187b560122ef2e8
Sep 15 07:35:40.376747 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:40.376648    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx" event={"ID":"95397f4f-11d5-4393-94ce-3d29ea930c5a","Type":"ContainerStarted","Data":"7e1b84b69213bc20e4ac29b5808b742c032e3a669e2e96ced6ad542e154d5a9a"}
Sep 15 07:35:40.376747 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:40.376705    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx" event={"ID":"95397f4f-11d5-4393-94ce-3d29ea930c5a","Type":"ContainerStarted","Data":"ee90f670116b45a57148a0574a7a3dafa9bf57d64aefea801187b560122ef2e8"}
Sep 15 07:35:40.376747 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:40.376728    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx"
Sep 15 07:35:40.395839 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:40.395785    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx" podStartSLOduration=1.395766966 podStartE2EDuration="1.395766966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:35:40.393757703 +0000 UTC m=+1044.940327827" watchObservedRunningTime="2026-09-15 07:35:40.395766966 +0000 UTC m=+1044.942337068"
Sep 15 07:35:42.330143 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:42.330106    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p"
Sep 15 07:35:42.330518 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:42.330291    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j"
Sep 15 07:35:44.344747 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:44.344711    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx"
Sep 15 07:35:51.384468 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:35:51.384434    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx"
Sep 15 07:36:04.581735 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.581699    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"]
Sep 15 07:36:04.586960 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.586938    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:04.590018 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.589987    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2735c5\"/\"kube-root-ca.crt\""
Sep 15 07:36:04.590133 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.590063    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2735c5\"/\"openshift-service-ca.crt\""
Sep 15 07:36:04.590898 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.590881    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2735c5\"/\"simulator-w0-2735c5-kserve-self-signed-certs\""
Sep 15 07:36:04.590996 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.590946    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2735c5\"/\"default-dockercfg-2jjvm\""
Sep 15 07:36:04.591058 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.591001    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"]
Sep 15 07:36:04.605788 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.605761    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-dshm\") pod \"simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") " pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:04.605917 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.605795    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-model-cache\") pod \"simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") " pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:04.605917 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.605830    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-home\") pod \"simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") " pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:04.605917 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.605882    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57z9l\" (UniqueName: \"kubernetes.io/projected/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-kube-api-access-57z9l\") pod \"simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") " pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:04.606027 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.605999    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-tls-certs\") pod \"simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") " pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:04.606096 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.606079    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-tmp-dir\") pod \"simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") " pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:04.706812 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.706780    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-dshm\") pod \"simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") " pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:04.706812 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.706817    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-model-cache\") pod \"simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") " pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:04.707061 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.706844    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-home\") pod \"simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") " pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:04.707061 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.706865    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57z9l\" (UniqueName: \"kubernetes.io/projected/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-kube-api-access-57z9l\") pod \"simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") " pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:04.707061 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.706893    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-tls-certs\") pod \"simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") " pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:04.707061 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.706931    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-tmp-dir\") pod \"simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") " pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:04.707368 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.707336    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-model-cache\") pod \"simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") " pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:04.707470 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.707367    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-home\") pod \"simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") " pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:04.707470 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.707380    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-tmp-dir\") pod \"simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") " pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:04.709074 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.709053    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-dshm\") pod \"simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") " pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:04.709339 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.709321    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-tls-certs\") pod \"simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") " pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:04.715771 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.715751    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57z9l\" (UniqueName: \"kubernetes.io/projected/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-kube-api-access-57z9l\") pod \"simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") " pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:04.897118 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:04.897033    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:05.168027 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:05.168002    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"]
Sep 15 07:36:05.169543 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:36:05.169519    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddff30f35_72cb_4fe8_b87d_8bc20401fd8d.slice/crio-2c5c660735ae1c8b33bc2cd6553cefe7af450af39d34c3799cea5581091e416a WatchSource:0}: Error finding container 2c5c660735ae1c8b33bc2cd6553cefe7af450af39d34c3799cea5581091e416a: Status 404 returned error can't find the container with id 2c5c660735ae1c8b33bc2cd6553cefe7af450af39d34c3799cea5581091e416a
Sep 15 07:36:05.510862 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:05.510827    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7" event={"ID":"dff30f35-72cb-4fe8-b87d-8bc20401fd8d","Type":"ContainerStarted","Data":"30fa8ce2066dad24915e2a757a4b60230c2d3bcea0902e591e4600e8e0a9690b"}
Sep 15 07:36:05.510862 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:05.510866    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7" event={"ID":"dff30f35-72cb-4fe8-b87d-8bc20401fd8d","Type":"ContainerStarted","Data":"2c5c660735ae1c8b33bc2cd6553cefe7af450af39d34c3799cea5581091e416a"}
Sep 15 07:36:05.527064 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:05.527009    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7" podStartSLOduration=1.526994714 podStartE2EDuration="1.526994714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:36:05.525088418 +0000 UTC m=+1070.071658540" watchObservedRunningTime="2026-09-15 07:36:05.526994714 +0000 UTC m=+1070.073564813"
Sep 15 07:36:07.468112 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:07.468074    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j"]
Sep 15 07:36:07.468589 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:07.468397    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j" podUID="9548c820-e25e-4a05-84ca-627208e624f0" containerName="payload-processing" containerID="cri-o://bbc99aef91686b0a0172105082a26a9ef49d85073b6473abf7e52363bda3ec65" gracePeriod=30
Sep 15 07:36:07.475763 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:07.475156    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p"]
Sep 15 07:36:07.475763 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:07.475395    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p" podUID="70c55d24-de21-4add-bdc3-1dfd87c0bd9c" containerName="payload-pre-processing" containerID="cri-o://b09a66b20ead14f63fb1f07b74e295f0c5228a5e637cfd72ba4319c460893773" gracePeriod=30
Sep 15 07:36:07.740418 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:07.740341    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j"
Sep 15 07:36:07.771460 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:07.771432    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p"
Sep 15 07:36:07.836570 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:07.836540    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70c55d24-de21-4add-bdc3-1dfd87c0bd9c-plugins-config-volume\") pod \"70c55d24-de21-4add-bdc3-1dfd87c0bd9c\" (UID: \"70c55d24-de21-4add-bdc3-1dfd87c0bd9c\") "
Sep 15 07:36:07.836772 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:07.836589    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9548c820-e25e-4a05-84ca-627208e624f0-plugins-config-volume\") pod \"9548c820-e25e-4a05-84ca-627208e624f0\" (UID: \"9548c820-e25e-4a05-84ca-627208e624f0\") "
Sep 15 07:36:07.836772 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:07.836659    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jj7l7\" (UniqueName: \"kubernetes.io/projected/9548c820-e25e-4a05-84ca-627208e624f0-kube-api-access-jj7l7\") pod \"9548c820-e25e-4a05-84ca-627208e624f0\" (UID: \"9548c820-e25e-4a05-84ca-627208e624f0\") "
Sep 15 07:36:07.836772 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:07.836731    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k7r5\" (UniqueName: \"kubernetes.io/projected/70c55d24-de21-4add-bdc3-1dfd87c0bd9c-kube-api-access-2k7r5\") pod \"70c55d24-de21-4add-bdc3-1dfd87c0bd9c\" (UID: \"70c55d24-de21-4add-bdc3-1dfd87c0bd9c\") "
Sep 15 07:36:07.836945 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:07.836916    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c55d24-de21-4add-bdc3-1dfd87c0bd9c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70c55d24-de21-4add-bdc3-1dfd87c0bd9c" (UID: "70c55d24-de21-4add-bdc3-1dfd87c0bd9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:36:07.837024 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:07.837004    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70c55d24-de21-4add-bdc3-1dfd87c0bd9c-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:36:07.837081 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:07.837053    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9548c820-e25e-4a05-84ca-627208e624f0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9548c820-e25e-4a05-84ca-627208e624f0" (UID: "9548c820-e25e-4a05-84ca-627208e624f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:36:07.838983 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:07.838962    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c55d24-de21-4add-bdc3-1dfd87c0bd9c-kube-api-access-2k7r5" (OuterVolumeSpecName: "kube-api-access-2k7r5") pod "70c55d24-de21-4add-bdc3-1dfd87c0bd9c" (UID: "70c55d24-de21-4add-bdc3-1dfd87c0bd9c"). InnerVolumeSpecName "kube-api-access-2k7r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:36:07.839075 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:07.839009    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9548c820-e25e-4a05-84ca-627208e624f0-kube-api-access-jj7l7" (OuterVolumeSpecName: "kube-api-access-jj7l7") pod "9548c820-e25e-4a05-84ca-627208e624f0" (UID: "9548c820-e25e-4a05-84ca-627208e624f0"). InnerVolumeSpecName "kube-api-access-jj7l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:36:07.938504 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:07.938474    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj7l7\" (UniqueName: \"kubernetes.io/projected/9548c820-e25e-4a05-84ca-627208e624f0-kube-api-access-jj7l7\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:36:07.938504 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:07.938503    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k7r5\" (UniqueName: \"kubernetes.io/projected/70c55d24-de21-4add-bdc3-1dfd87c0bd9c-kube-api-access-2k7r5\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:36:07.938640 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:07.938517    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9548c820-e25e-4a05-84ca-627208e624f0-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:36:08.528661 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:08.528626    2571 generic.go:356] "Generic (PLEG): container finished" podID="70c55d24-de21-4add-bdc3-1dfd87c0bd9c" containerID="b09a66b20ead14f63fb1f07b74e295f0c5228a5e637cfd72ba4319c460893773" exitCode=0
Sep 15 07:36:08.529116 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:08.528673    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p"
Sep 15 07:36:08.529116 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:08.528711    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p" event={"ID":"70c55d24-de21-4add-bdc3-1dfd87c0bd9c","Type":"ContainerDied","Data":"b09a66b20ead14f63fb1f07b74e295f0c5228a5e637cfd72ba4319c460893773"}
Sep 15 07:36:08.529116 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:08.528774    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p" event={"ID":"70c55d24-de21-4add-bdc3-1dfd87c0bd9c","Type":"ContainerDied","Data":"9938c543c7e2469d577c4ccc1db6ab9c001d667573cde23bdc55ef1985ed5b07"}
Sep 15 07:36:08.529116 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:08.528803    2571 scope.go:117] "RemoveContainer" containerID="b09a66b20ead14f63fb1f07b74e295f0c5228a5e637cfd72ba4319c460893773"
Sep 15 07:36:08.530136 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:08.530113    2571 generic.go:356] "Generic (PLEG): container finished" podID="9548c820-e25e-4a05-84ca-627208e624f0" containerID="bbc99aef91686b0a0172105082a26a9ef49d85073b6473abf7e52363bda3ec65" exitCode=0
Sep 15 07:36:08.530211 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:08.530159    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j"
Sep 15 07:36:08.530299 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:08.530202    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j" event={"ID":"9548c820-e25e-4a05-84ca-627208e624f0","Type":"ContainerDied","Data":"bbc99aef91686b0a0172105082a26a9ef49d85073b6473abf7e52363bda3ec65"}
Sep 15 07:36:08.530299 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:08.530283    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j" event={"ID":"9548c820-e25e-4a05-84ca-627208e624f0","Type":"ContainerDied","Data":"562e4df6b04a05464861d78d09fb2c59ba0019f55d85e51da59a96c7d5acf84d"}
Sep 15 07:36:08.538194 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:08.538176    2571 scope.go:117] "RemoveContainer" containerID="b09a66b20ead14f63fb1f07b74e295f0c5228a5e637cfd72ba4319c460893773"
Sep 15 07:36:08.538539 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:36:08.538522    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b09a66b20ead14f63fb1f07b74e295f0c5228a5e637cfd72ba4319c460893773\": container with ID starting with b09a66b20ead14f63fb1f07b74e295f0c5228a5e637cfd72ba4319c460893773 not found: ID does not exist" containerID="b09a66b20ead14f63fb1f07b74e295f0c5228a5e637cfd72ba4319c460893773"
Sep 15 07:36:08.538590 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:08.538548    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b09a66b20ead14f63fb1f07b74e295f0c5228a5e637cfd72ba4319c460893773"} err="failed to get container status \"b09a66b20ead14f63fb1f07b74e295f0c5228a5e637cfd72ba4319c460893773\": rpc error: code = NotFound desc = could not find container \"b09a66b20ead14f63fb1f07b74e295f0c5228a5e637cfd72ba4319c460893773\": container with ID starting with b09a66b20ead14f63fb1f07b74e295f0c5228a5e637cfd72ba4319c460893773 not found: ID does not exist"
Sep 15 07:36:08.538590 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:08.538564    2571 scope.go:117] "RemoveContainer" containerID="bbc99aef91686b0a0172105082a26a9ef49d85073b6473abf7e52363bda3ec65"
Sep 15 07:36:08.546547 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:08.546511    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p"]
Sep 15 07:36:08.547716 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:08.547694    2571 scope.go:117] "RemoveContainer" containerID="bbc99aef91686b0a0172105082a26a9ef49d85073b6473abf7e52363bda3ec65"
Sep 15 07:36:08.548032 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:36:08.548009    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbc99aef91686b0a0172105082a26a9ef49d85073b6473abf7e52363bda3ec65\": container with ID starting with bbc99aef91686b0a0172105082a26a9ef49d85073b6473abf7e52363bda3ec65 not found: ID does not exist" containerID="bbc99aef91686b0a0172105082a26a9ef49d85073b6473abf7e52363bda3ec65"
Sep 15 07:36:08.548099 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:08.548040    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc99aef91686b0a0172105082a26a9ef49d85073b6473abf7e52363bda3ec65"} err="failed to get container status \"bbc99aef91686b0a0172105082a26a9ef49d85073b6473abf7e52363bda3ec65\": rpc error: code = NotFound desc = could not find container \"bbc99aef91686b0a0172105082a26a9ef49d85073b6473abf7e52363bda3ec65\": container with ID starting with bbc99aef91686b0a0172105082a26a9ef49d85073b6473abf7e52363bda3ec65 not found: ID does not exist"
Sep 15 07:36:08.548594 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:08.548574    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e8faa2dc-68b7574b66-9fv6p"]
Sep 15 07:36:08.557002 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:08.556979    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j"]
Sep 15 07:36:08.562860 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:08.562840    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e8faa2dc-d7675fcc5-st86j"]
Sep 15 07:36:09.910301 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:09.910265    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70c55d24-de21-4add-bdc3-1dfd87c0bd9c" path="/var/lib/kubelet/pods/70c55d24-de21-4add-bdc3-1dfd87c0bd9c/volumes"
Sep 15 07:36:09.910676 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:09.910637    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9548c820-e25e-4a05-84ca-627208e624f0" path="/var/lib/kubelet/pods/9548c820-e25e-4a05-84ca-627208e624f0/volumes"
Sep 15 07:36:13.650495 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.650458    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl"]
Sep 15 07:36:13.650912 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.650898    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70c55d24-de21-4add-bdc3-1dfd87c0bd9c" containerName="payload-pre-processing"
Sep 15 07:36:13.650912 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.650913    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="70c55d24-de21-4add-bdc3-1dfd87c0bd9c" containerName="payload-pre-processing"
Sep 15 07:36:13.650997 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.650935    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9548c820-e25e-4a05-84ca-627208e624f0" containerName="payload-processing"
Sep 15 07:36:13.650997 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.650940    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9548c820-e25e-4a05-84ca-627208e624f0" containerName="payload-processing"
Sep 15 07:36:13.651073 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.651004    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9548c820-e25e-4a05-84ca-627208e624f0" containerName="payload-processing"
Sep 15 07:36:13.651073 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.651017    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="70c55d24-de21-4add-bdc3-1dfd87c0bd9c" containerName="payload-pre-processing"
Sep 15 07:36:13.654119 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.654100    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl"
Sep 15 07:36:13.669826 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.669790    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4e9b0a83\""
Sep 15 07:36:13.669933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.669855    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-4e9b0a83-dockercfg-kvq46\""
Sep 15 07:36:13.673747 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.673721    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl"]
Sep 15 07:36:13.693804 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.693776    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcdp6\" (UniqueName: \"kubernetes.io/projected/a87b975f-1fd5-41c3-b302-c81faf6c66e0-kube-api-access-fcdp6\") pod \"payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl\" (UID: \"a87b975f-1fd5-41c3-b302-c81faf6c66e0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl"
Sep 15 07:36:13.693934 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.693835    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87b975f-1fd5-41c3-b302-c81faf6c66e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl\" (UID: \"a87b975f-1fd5-41c3-b302-c81faf6c66e0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl"
Sep 15 07:36:13.710032 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.709998    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9"]
Sep 15 07:36:13.713590 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.713572    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9"
Sep 15 07:36:13.737461 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.737435    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9"]
Sep 15 07:36:13.795168 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.795139    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa9dc68d-b099-4d1e-80d7-88800f7ab07c-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9\" (UID: \"fa9dc68d-b099-4d1e-80d7-88800f7ab07c\") " pod="openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9"
Sep 15 07:36:13.795363 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.795238    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcdp6\" (UniqueName: \"kubernetes.io/projected/a87b975f-1fd5-41c3-b302-c81faf6c66e0-kube-api-access-fcdp6\") pod \"payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl\" (UID: \"a87b975f-1fd5-41c3-b302-c81faf6c66e0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl"
Sep 15 07:36:13.795363 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.795334    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87b975f-1fd5-41c3-b302-c81faf6c66e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl\" (UID: \"a87b975f-1fd5-41c3-b302-c81faf6c66e0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl"
Sep 15 07:36:13.795363 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.795355    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thb9p\" (UniqueName: \"kubernetes.io/projected/fa9dc68d-b099-4d1e-80d7-88800f7ab07c-kube-api-access-thb9p\") pod \"payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9\" (UID: \"fa9dc68d-b099-4d1e-80d7-88800f7ab07c\") " pod="openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9"
Sep 15 07:36:13.796025 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.796007    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87b975f-1fd5-41c3-b302-c81faf6c66e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl\" (UID: \"a87b975f-1fd5-41c3-b302-c81faf6c66e0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl"
Sep 15 07:36:13.805652 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.805629    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcdp6\" (UniqueName: \"kubernetes.io/projected/a87b975f-1fd5-41c3-b302-c81faf6c66e0-kube-api-access-fcdp6\") pod \"payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl\" (UID: \"a87b975f-1fd5-41c3-b302-c81faf6c66e0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl"
Sep 15 07:36:13.896410 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.896373    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thb9p\" (UniqueName: \"kubernetes.io/projected/fa9dc68d-b099-4d1e-80d7-88800f7ab07c-kube-api-access-thb9p\") pod \"payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9\" (UID: \"fa9dc68d-b099-4d1e-80d7-88800f7ab07c\") " pod="openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9"
Sep 15 07:36:13.896572 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.896439    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa9dc68d-b099-4d1e-80d7-88800f7ab07c-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9\" (UID: \"fa9dc68d-b099-4d1e-80d7-88800f7ab07c\") " pod="openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9"
Sep 15 07:36:13.897165 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.897137    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa9dc68d-b099-4d1e-80d7-88800f7ab07c-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9\" (UID: \"fa9dc68d-b099-4d1e-80d7-88800f7ab07c\") " pod="openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9"
Sep 15 07:36:13.904740 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.904679    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thb9p\" (UniqueName: \"kubernetes.io/projected/fa9dc68d-b099-4d1e-80d7-88800f7ab07c-kube-api-access-thb9p\") pod \"payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9\" (UID: \"fa9dc68d-b099-4d1e-80d7-88800f7ab07c\") " pod="openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9"
Sep 15 07:36:13.964288 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:13.964239    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl"
Sep 15 07:36:14.024781 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:14.023612    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9"
Sep 15 07:36:14.234335 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:14.234311    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl"]
Sep 15 07:36:14.236097 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:36:14.236072    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87b975f_1fd5_41c3_b302_c81faf6c66e0.slice/crio-dfa9aae6805abb976f4f06107aea30787bc389fd0b5c1866a93d0ca6891030ae WatchSource:0}: Error finding container dfa9aae6805abb976f4f06107aea30787bc389fd0b5c1866a93d0ca6891030ae: Status 404 returned error can't find the container with id dfa9aae6805abb976f4f06107aea30787bc389fd0b5c1866a93d0ca6891030ae
Sep 15 07:36:14.300372 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:14.300348    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9"]
Sep 15 07:36:14.302441 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:36:14.302418    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa9dc68d_b099_4d1e_80d7_88800f7ab07c.slice/crio-f7fe6cde989a2c2c0b0252544166d94569534da51bb580df3753e052cf42c957 WatchSource:0}: Error finding container f7fe6cde989a2c2c0b0252544166d94569534da51bb580df3753e052cf42c957: Status 404 returned error can't find the container with id f7fe6cde989a2c2c0b0252544166d94569534da51bb580df3753e052cf42c957
Sep 15 07:36:14.568881 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:14.568792    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9" event={"ID":"fa9dc68d-b099-4d1e-80d7-88800f7ab07c","Type":"ContainerStarted","Data":"8a44dbde2be62a593e6eab54a62e4fc0e144ff953114c4f3a606b7c885158439"}
Sep 15 07:36:14.568881 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:14.568850    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9" event={"ID":"fa9dc68d-b099-4d1e-80d7-88800f7ab07c","Type":"ContainerStarted","Data":"f7fe6cde989a2c2c0b0252544166d94569534da51bb580df3753e052cf42c957"}
Sep 15 07:36:14.568881 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:14.568880    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9"
Sep 15 07:36:14.570556 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:14.570532    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl" event={"ID":"a87b975f-1fd5-41c3-b302-c81faf6c66e0","Type":"ContainerStarted","Data":"b27cede09ddbff903175700b0372b8c7bc53899d20e788e4faab8e666826a7b9"}
Sep 15 07:36:14.570679 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:14.570581    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl"
Sep 15 07:36:14.570679 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:14.570605    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl" event={"ID":"a87b975f-1fd5-41c3-b302-c81faf6c66e0","Type":"ContainerStarted","Data":"dfa9aae6805abb976f4f06107aea30787bc389fd0b5c1866a93d0ca6891030ae"}
Sep 15 07:36:14.592451 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:14.592403    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9" podStartSLOduration=1.5923906799999998 podStartE2EDuration="1.59239068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:36:14.590710871 +0000 UTC m=+1079.137280989" watchObservedRunningTime="2026-09-15 07:36:14.59239068 +0000 UTC m=+1079.138960778"
Sep 15 07:36:14.609527 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:14.609473    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl" podStartSLOduration=1.6094599139999999 podStartE2EDuration="1.609459914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:36:14.607213982 +0000 UTC m=+1079.153784112" watchObservedRunningTime="2026-09-15 07:36:14.609459914 +0000 UTC m=+1079.156030013"
Sep 15 07:36:14.897496 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:14.897399    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:14.897496 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:14.897447    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:14.910476 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:14.910442    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:15.587421 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:15.587387    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:36:25.578400 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:25.578368    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9"
Sep 15 07:36:25.578957 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:25.578438    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl"
Sep 15 07:36:31.588113 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.588075    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"]
Sep 15 07:36:31.593584 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.593563    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:31.595966 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.595932    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f56165\"/\"simulator-w4-f56165-kserve-self-signed-certs\""
Sep 15 07:36:31.597153 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.597107    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f56165\"/\"openshift-service-ca.crt\""
Sep 15 07:36:31.597292 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.597197    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f56165\"/\"default-dockercfg-p295w\""
Sep 15 07:36:31.597292 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.597107    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f56165\"/\"kube-root-ca.crt\""
Sep 15 07:36:31.600907 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.600887    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"]
Sep 15 07:36:31.676789 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.676754    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-model-cache\") pod \"simulator-w4-f56165-kserve-7f4d9f684b-2rpt9\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") " pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:31.676970 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.676861    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmkzl\" (UniqueName: \"kubernetes.io/projected/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-kube-api-access-vmkzl\") pod \"simulator-w4-f56165-kserve-7f4d9f684b-2rpt9\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") " pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:31.676970 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.676942    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-dshm\") pod \"simulator-w4-f56165-kserve-7f4d9f684b-2rpt9\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") " pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:31.677084 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.677011    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-tmp-dir\") pod \"simulator-w4-f56165-kserve-7f4d9f684b-2rpt9\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") " pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:31.677084 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.677044    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-home\") pod \"simulator-w4-f56165-kserve-7f4d9f684b-2rpt9\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") " pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:31.677167 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.677093    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-tls-certs\") pod \"simulator-w4-f56165-kserve-7f4d9f684b-2rpt9\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") " pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:31.777836 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.777777    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-tls-certs\") pod \"simulator-w4-f56165-kserve-7f4d9f684b-2rpt9\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") " pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:31.778026 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.777867    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-model-cache\") pod \"simulator-w4-f56165-kserve-7f4d9f684b-2rpt9\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") " pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:31.778026 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.777914    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmkzl\" (UniqueName: \"kubernetes.io/projected/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-kube-api-access-vmkzl\") pod \"simulator-w4-f56165-kserve-7f4d9f684b-2rpt9\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") " pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:31.778026 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.777953    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-dshm\") pod \"simulator-w4-f56165-kserve-7f4d9f684b-2rpt9\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") " pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:31.778026 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.777984    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-tmp-dir\") pod \"simulator-w4-f56165-kserve-7f4d9f684b-2rpt9\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") " pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:31.778026 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.778000    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-home\") pod \"simulator-w4-f56165-kserve-7f4d9f684b-2rpt9\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") " pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:31.778402 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.778375    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-model-cache\") pod \"simulator-w4-f56165-kserve-7f4d9f684b-2rpt9\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") " pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:31.778402 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.778395    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-tmp-dir\") pod \"simulator-w4-f56165-kserve-7f4d9f684b-2rpt9\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") " pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:31.778497 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.778444    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-home\") pod \"simulator-w4-f56165-kserve-7f4d9f684b-2rpt9\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") " pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:31.780177 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.780155    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-dshm\") pod \"simulator-w4-f56165-kserve-7f4d9f684b-2rpt9\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") " pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:31.780237 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.780218    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-tls-certs\") pod \"simulator-w4-f56165-kserve-7f4d9f684b-2rpt9\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") " pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:31.786291 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.786271    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmkzl\" (UniqueName: \"kubernetes.io/projected/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-kube-api-access-vmkzl\") pod \"simulator-w4-f56165-kserve-7f4d9f684b-2rpt9\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") " pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:31.904848 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:31.904741    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:32.177232 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:32.177205    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"]
Sep 15 07:36:32.178246 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:36:32.178210    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f8b4cd6_013e_4a5d_ba1e_f39130dad902.slice/crio-b3ac1f0aa4c2197b2552fd390c93e318d51fe5a32286c423750639e66c43c9a1 WatchSource:0}: Error finding container b3ac1f0aa4c2197b2552fd390c93e318d51fe5a32286c423750639e66c43c9a1: Status 404 returned error can't find the container with id b3ac1f0aa4c2197b2552fd390c93e318d51fe5a32286c423750639e66c43c9a1
Sep 15 07:36:32.676582 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:32.676544    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9" event={"ID":"0f8b4cd6-013e-4a5d-ba1e-f39130dad902","Type":"ContainerStarted","Data":"9d0dae549f0ef10fd888a37817d77e46a57c09f101440960962407f4265d39db"}
Sep 15 07:36:32.676979 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:32.676598    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9" event={"ID":"0f8b4cd6-013e-4a5d-ba1e-f39130dad902","Type":"ContainerStarted","Data":"b3ac1f0aa4c2197b2552fd390c93e318d51fe5a32286c423750639e66c43c9a1"}
Sep 15 07:36:32.701954 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:32.701910    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9" podStartSLOduration=1.701892822 podStartE2EDuration="1.701892822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:36:32.700684605 +0000 UTC m=+1097.247254723" watchObservedRunningTime="2026-09-15 07:36:32.701892822 +0000 UTC m=+1097.248462922"
Sep 15 07:36:41.910922 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:41.910887    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:41.911298 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:41.910931    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:41.918921 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:41.918893    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:42.726347 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:42.726307    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9"]
Sep 15 07:36:42.726591 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:42.726549    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9" podUID="fa9dc68d-b099-4d1e-80d7-88800f7ab07c" containerName="payload-processing" containerID="cri-o://8a44dbde2be62a593e6eab54a62e4fc0e144ff953114c4f3a606b7c885158439" gracePeriod=30
Sep 15 07:36:42.736068 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:42.736041    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl"]
Sep 15 07:36:42.736395 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:42.736365    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl" podUID="a87b975f-1fd5-41c3-b302-c81faf6c66e0" containerName="payload-pre-processing" containerID="cri-o://b27cede09ddbff903175700b0372b8c7bc53899d20e788e4faab8e666826a7b9" gracePeriod=30
Sep 15 07:36:42.748381 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:42.748334    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:36:43.030303 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.030275    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9"
Sep 15 07:36:43.034574 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.034553    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl"
Sep 15 07:36:43.093742 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.093708    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87b975f-1fd5-41c3-b302-c81faf6c66e0-plugins-config-volume\") pod \"a87b975f-1fd5-41c3-b302-c81faf6c66e0\" (UID: \"a87b975f-1fd5-41c3-b302-c81faf6c66e0\") "
Sep 15 07:36:43.093900 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.093795    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa9dc68d-b099-4d1e-80d7-88800f7ab07c-plugins-config-volume\") pod \"fa9dc68d-b099-4d1e-80d7-88800f7ab07c\" (UID: \"fa9dc68d-b099-4d1e-80d7-88800f7ab07c\") "
Sep 15 07:36:43.093900 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.093846    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcdp6\" (UniqueName: \"kubernetes.io/projected/a87b975f-1fd5-41c3-b302-c81faf6c66e0-kube-api-access-fcdp6\") pod \"a87b975f-1fd5-41c3-b302-c81faf6c66e0\" (UID: \"a87b975f-1fd5-41c3-b302-c81faf6c66e0\") "
Sep 15 07:36:43.093900 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.093866    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thb9p\" (UniqueName: \"kubernetes.io/projected/fa9dc68d-b099-4d1e-80d7-88800f7ab07c-kube-api-access-thb9p\") pod \"fa9dc68d-b099-4d1e-80d7-88800f7ab07c\" (UID: \"fa9dc68d-b099-4d1e-80d7-88800f7ab07c\") "
Sep 15 07:36:43.094149 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.094121    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a87b975f-1fd5-41c3-b302-c81faf6c66e0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a87b975f-1fd5-41c3-b302-c81faf6c66e0" (UID: "a87b975f-1fd5-41c3-b302-c81faf6c66e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:36:43.094218 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.094200    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa9dc68d-b099-4d1e-80d7-88800f7ab07c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fa9dc68d-b099-4d1e-80d7-88800f7ab07c" (UID: "fa9dc68d-b099-4d1e-80d7-88800f7ab07c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:36:43.095897 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.095876    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa9dc68d-b099-4d1e-80d7-88800f7ab07c-kube-api-access-thb9p" (OuterVolumeSpecName: "kube-api-access-thb9p") pod "fa9dc68d-b099-4d1e-80d7-88800f7ab07c" (UID: "fa9dc68d-b099-4d1e-80d7-88800f7ab07c"). InnerVolumeSpecName "kube-api-access-thb9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:36:43.095972 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.095919    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a87b975f-1fd5-41c3-b302-c81faf6c66e0-kube-api-access-fcdp6" (OuterVolumeSpecName: "kube-api-access-fcdp6") pod "a87b975f-1fd5-41c3-b302-c81faf6c66e0" (UID: "a87b975f-1fd5-41c3-b302-c81faf6c66e0"). InnerVolumeSpecName "kube-api-access-fcdp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:36:43.194853 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.194816    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa9dc68d-b099-4d1e-80d7-88800f7ab07c-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:36:43.194853 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.194849    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcdp6\" (UniqueName: \"kubernetes.io/projected/a87b975f-1fd5-41c3-b302-c81faf6c66e0-kube-api-access-fcdp6\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:36:43.194853 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.194858    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thb9p\" (UniqueName: \"kubernetes.io/projected/fa9dc68d-b099-4d1e-80d7-88800f7ab07c-kube-api-access-thb9p\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:36:43.195095 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.194867    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87b975f-1fd5-41c3-b302-c81faf6c66e0-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:36:43.736175 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.736131    2571 generic.go:356] "Generic (PLEG): container finished" podID="fa9dc68d-b099-4d1e-80d7-88800f7ab07c" containerID="8a44dbde2be62a593e6eab54a62e4fc0e144ff953114c4f3a606b7c885158439" exitCode=0
Sep 15 07:36:43.736388 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.736235    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9"
Sep 15 07:36:43.736388 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.736299    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9" event={"ID":"fa9dc68d-b099-4d1e-80d7-88800f7ab07c","Type":"ContainerDied","Data":"8a44dbde2be62a593e6eab54a62e4fc0e144ff953114c4f3a606b7c885158439"}
Sep 15 07:36:43.736388 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.736370    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9" event={"ID":"fa9dc68d-b099-4d1e-80d7-88800f7ab07c","Type":"ContainerDied","Data":"f7fe6cde989a2c2c0b0252544166d94569534da51bb580df3753e052cf42c957"}
Sep 15 07:36:43.736573 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.736400    2571 scope.go:117] "RemoveContainer" containerID="8a44dbde2be62a593e6eab54a62e4fc0e144ff953114c4f3a606b7c885158439"
Sep 15 07:36:43.738320 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.738293    2571 generic.go:356] "Generic (PLEG): container finished" podID="a87b975f-1fd5-41c3-b302-c81faf6c66e0" containerID="b27cede09ddbff903175700b0372b8c7bc53899d20e788e4faab8e666826a7b9" exitCode=0
Sep 15 07:36:43.738466 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.738332    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl"
Sep 15 07:36:43.738466 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.738360    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl" event={"ID":"a87b975f-1fd5-41c3-b302-c81faf6c66e0","Type":"ContainerDied","Data":"b27cede09ddbff903175700b0372b8c7bc53899d20e788e4faab8e666826a7b9"}
Sep 15 07:36:43.738466 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.738402    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl" event={"ID":"a87b975f-1fd5-41c3-b302-c81faf6c66e0","Type":"ContainerDied","Data":"dfa9aae6805abb976f4f06107aea30787bc389fd0b5c1866a93d0ca6891030ae"}
Sep 15 07:36:43.748752 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.748730    2571 scope.go:117] "RemoveContainer" containerID="8a44dbde2be62a593e6eab54a62e4fc0e144ff953114c4f3a606b7c885158439"
Sep 15 07:36:43.749081 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:36:43.749054    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a44dbde2be62a593e6eab54a62e4fc0e144ff953114c4f3a606b7c885158439\": container with ID starting with 8a44dbde2be62a593e6eab54a62e4fc0e144ff953114c4f3a606b7c885158439 not found: ID does not exist" containerID="8a44dbde2be62a593e6eab54a62e4fc0e144ff953114c4f3a606b7c885158439"
Sep 15 07:36:43.749143 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.749093    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a44dbde2be62a593e6eab54a62e4fc0e144ff953114c4f3a606b7c885158439"} err="failed to get container status \"8a44dbde2be62a593e6eab54a62e4fc0e144ff953114c4f3a606b7c885158439\": rpc error: code = NotFound desc = could not find container \"8a44dbde2be62a593e6eab54a62e4fc0e144ff953114c4f3a606b7c885158439\": container with ID starting with 8a44dbde2be62a593e6eab54a62e4fc0e144ff953114c4f3a606b7c885158439 not found: ID does not exist"
Sep 15 07:36:43.749143 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.749117    2571 scope.go:117] "RemoveContainer" containerID="b27cede09ddbff903175700b0372b8c7bc53899d20e788e4faab8e666826a7b9"
Sep 15 07:36:43.760334 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.760313    2571 scope.go:117] "RemoveContainer" containerID="b27cede09ddbff903175700b0372b8c7bc53899d20e788e4faab8e666826a7b9"
Sep 15 07:36:43.760728 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:36:43.760703    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b27cede09ddbff903175700b0372b8c7bc53899d20e788e4faab8e666826a7b9\": container with ID starting with b27cede09ddbff903175700b0372b8c7bc53899d20e788e4faab8e666826a7b9 not found: ID does not exist" containerID="b27cede09ddbff903175700b0372b8c7bc53899d20e788e4faab8e666826a7b9"
Sep 15 07:36:43.760796 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.760740    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b27cede09ddbff903175700b0372b8c7bc53899d20e788e4faab8e666826a7b9"} err="failed to get container status \"b27cede09ddbff903175700b0372b8c7bc53899d20e788e4faab8e666826a7b9\": rpc error: code = NotFound desc = could not find container \"b27cede09ddbff903175700b0372b8c7bc53899d20e788e4faab8e666826a7b9\": container with ID starting with b27cede09ddbff903175700b0372b8c7bc53899d20e788e4faab8e666826a7b9 not found: ID does not exist"
Sep 15 07:36:43.773941 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.773897    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9"]
Sep 15 07:36:43.779788 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.779758    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4e9b0a83-598d55f6bb-pksj9"]
Sep 15 07:36:43.792819 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.792788    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl"]
Sep 15 07:36:43.798189 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.798164    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4e9b0a83-64c5cc8c7d-qxdgl"]
Sep 15 07:36:43.910089 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.910056    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a87b975f-1fd5-41c3-b302-c81faf6c66e0" path="/var/lib/kubelet/pods/a87b975f-1fd5-41c3-b302-c81faf6c66e0/volumes"
Sep 15 07:36:43.910521 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:43.910504    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa9dc68d-b099-4d1e-80d7-88800f7ab07c" path="/var/lib/kubelet/pods/fa9dc68d-b099-4d1e-80d7-88800f7ab07c/volumes"
Sep 15 07:36:48.972420 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:48.972384    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f"]
Sep 15 07:36:48.973014 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:48.972993    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa9dc68d-b099-4d1e-80d7-88800f7ab07c" containerName="payload-processing"
Sep 15 07:36:48.973104 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:48.973017    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa9dc68d-b099-4d1e-80d7-88800f7ab07c" containerName="payload-processing"
Sep 15 07:36:48.973104 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:48.973064    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87b975f-1fd5-41c3-b302-c81faf6c66e0" containerName="payload-pre-processing"
Sep 15 07:36:48.973104 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:48.973073    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87b975f-1fd5-41c3-b302-c81faf6c66e0" containerName="payload-pre-processing"
Sep 15 07:36:48.973313 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:48.973207    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa9dc68d-b099-4d1e-80d7-88800f7ab07c" containerName="payload-processing"
Sep 15 07:36:48.973313 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:48.973222    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87b975f-1fd5-41c3-b302-c81faf6c66e0" containerName="payload-pre-processing"
Sep 15 07:36:48.977922 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:48.977900    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f"
Sep 15 07:36:48.982002 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:48.981979    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b63bda62\""
Sep 15 07:36:48.982088 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:48.981979    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b63bda62-dockercfg-8tb4d\""
Sep 15 07:36:48.987168 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:48.987148    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f"]
Sep 15 07:36:48.994514 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:48.994488    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7"]
Sep 15 07:36:48.998957 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:48.998873    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7"
Sep 15 07:36:49.007991 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.007968    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7"]
Sep 15 07:36:49.051141 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.051105    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca-plugins-config-volume\") pod \"payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7\" (UID: \"2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca\") " pod="openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7"
Sep 15 07:36:49.051325 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.051170    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plbp4\" (UniqueName: \"kubernetes.io/projected/2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca-kube-api-access-plbp4\") pod \"payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7\" (UID: \"2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca\") " pod="openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7"
Sep 15 07:36:49.051325 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.051303    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7309b062-9756-474a-b633-1a9399ad813d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f\" (UID: \"7309b062-9756-474a-b633-1a9399ad813d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f"
Sep 15 07:36:49.051413 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.051394    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjz4p\" (UniqueName: \"kubernetes.io/projected/7309b062-9756-474a-b633-1a9399ad813d-kube-api-access-zjz4p\") pod \"payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f\" (UID: \"7309b062-9756-474a-b633-1a9399ad813d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f"
Sep 15 07:36:49.152749 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.152708    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca-plugins-config-volume\") pod \"payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7\" (UID: \"2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca\") " pod="openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7"
Sep 15 07:36:49.152926 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.152790    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plbp4\" (UniqueName: \"kubernetes.io/projected/2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca-kube-api-access-plbp4\") pod \"payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7\" (UID: \"2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca\") " pod="openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7"
Sep 15 07:36:49.152926 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.152848    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7309b062-9756-474a-b633-1a9399ad813d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f\" (UID: \"7309b062-9756-474a-b633-1a9399ad813d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f"
Sep 15 07:36:49.152926 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.152900    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjz4p\" (UniqueName: \"kubernetes.io/projected/7309b062-9756-474a-b633-1a9399ad813d-kube-api-access-zjz4p\") pod \"payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f\" (UID: \"7309b062-9756-474a-b633-1a9399ad813d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f"
Sep 15 07:36:49.153457 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.153437    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca-plugins-config-volume\") pod \"payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7\" (UID: \"2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca\") " pod="openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7"
Sep 15 07:36:49.153655 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.153636    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7309b062-9756-474a-b633-1a9399ad813d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f\" (UID: \"7309b062-9756-474a-b633-1a9399ad813d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f"
Sep 15 07:36:49.161273 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.161238    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plbp4\" (UniqueName: \"kubernetes.io/projected/2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca-kube-api-access-plbp4\") pod \"payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7\" (UID: \"2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca\") " pod="openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7"
Sep 15 07:36:49.161421 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.161401    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjz4p\" (UniqueName: \"kubernetes.io/projected/7309b062-9756-474a-b633-1a9399ad813d-kube-api-access-zjz4p\") pod \"payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f\" (UID: \"7309b062-9756-474a-b633-1a9399ad813d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f"
Sep 15 07:36:49.289327 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.289209    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f"
Sep 15 07:36:49.310793 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.310759    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7"
Sep 15 07:36:49.565357 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.565332    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f"]
Sep 15 07:36:49.567382 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:36:49.567355    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7309b062_9756_474a_b633_1a9399ad813d.slice/crio-0c168e8adb6b2da73ec9a29c6d1a6cc147e48dd92f8b5b51cf058852cdadd971 WatchSource:0}: Error finding container 0c168e8adb6b2da73ec9a29c6d1a6cc147e48dd92f8b5b51cf058852cdadd971: Status 404 returned error can't find the container with id 0c168e8adb6b2da73ec9a29c6d1a6cc147e48dd92f8b5b51cf058852cdadd971
Sep 15 07:36:49.594310 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.594288    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7"]
Sep 15 07:36:49.598438 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:36:49.598412    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de8f3e9_bde1_4d9c_8dd3_0ba86ca3c8ca.slice/crio-1b7f19133fb42d8df5d7fc57ad1b19f7ef3cf29ee0d62b6aece8de8f19105a39 WatchSource:0}: Error finding container 1b7f19133fb42d8df5d7fc57ad1b19f7ef3cf29ee0d62b6aece8de8f19105a39: Status 404 returned error can't find the container with id 1b7f19133fb42d8df5d7fc57ad1b19f7ef3cf29ee0d62b6aece8de8f19105a39
Sep 15 07:36:49.772897 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.772851    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7" event={"ID":"2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca","Type":"ContainerStarted","Data":"64cbc35229cb2ade462c2324e911b21a2e38c48ff40a59316298b5ff81441962"}
Sep 15 07:36:49.773087 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.772922    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7" event={"ID":"2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca","Type":"ContainerStarted","Data":"1b7f19133fb42d8df5d7fc57ad1b19f7ef3cf29ee0d62b6aece8de8f19105a39"}
Sep 15 07:36:49.773087 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.772953    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7"
Sep 15 07:36:49.774515 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.774488    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f" event={"ID":"7309b062-9756-474a-b633-1a9399ad813d","Type":"ContainerStarted","Data":"5f14e6c04782e5115ea1ad0c2a97e1991cfbe5ef3ffbefbd6eb69d83f6ca8502"}
Sep 15 07:36:49.774625 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.774540    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f" event={"ID":"7309b062-9756-474a-b633-1a9399ad813d","Type":"ContainerStarted","Data":"0c168e8adb6b2da73ec9a29c6d1a6cc147e48dd92f8b5b51cf058852cdadd971"}
Sep 15 07:36:49.774625 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.774569    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f"
Sep 15 07:36:49.790779 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:49.790737    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7" podStartSLOduration=1.790722765 podStartE2EDuration="1.790722765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:36:49.788610705 +0000 UTC m=+1114.335180816" watchObservedRunningTime="2026-09-15 07:36:49.790722765 +0000 UTC m=+1114.337292935"
Sep 15 07:36:50.792477 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.792414    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f" podStartSLOduration=2.792391585 podStartE2EDuration="2.792391585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:36:49.815269607 +0000 UTC m=+1114.361839703" watchObservedRunningTime="2026-09-15 07:36:50.792391585 +0000 UTC m=+1115.338961685"
Sep 15 07:36:50.793610 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.793587    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"]
Sep 15 07:36:50.797467 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.797448    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:36:50.799890 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.799862    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-f33ec8\"/\"openshift-service-ca.crt\""
Sep 15 07:36:50.800136 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.800109    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-f33ec8\"/\"default-dockercfg-n77fv\""
Sep 15 07:36:50.800724 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.800706    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-f33ec8\"/\"kube-root-ca.crt\""
Sep 15 07:36:50.800858 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.800839    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-f33ec8\"/\"autores-model-f33ec8-kserve-self-signed-certs\""
Sep 15 07:36:50.806184 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.806159    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"]
Sep 15 07:36:50.869003 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.868970    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02475b45-8fb4-439f-a688-989710da1a50-tls-certs\") pod \"autores-model-f33ec8-kserve-787964f899-vvtvr\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") " pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:36:50.869188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.869077    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-model-cache\") pod \"autores-model-f33ec8-kserve-787964f899-vvtvr\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") " pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:36:50.869188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.869130    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-dshm\") pod \"autores-model-f33ec8-kserve-787964f899-vvtvr\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") " pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:36:50.869188 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.869166    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-home\") pod \"autores-model-f33ec8-kserve-787964f899-vvtvr\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") " pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:36:50.869391 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.869234    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-tmp-dir\") pod \"autores-model-f33ec8-kserve-787964f899-vvtvr\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") " pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:36:50.869391 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.869325    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8s8r\" (UniqueName: \"kubernetes.io/projected/02475b45-8fb4-439f-a688-989710da1a50-kube-api-access-q8s8r\") pod \"autores-model-f33ec8-kserve-787964f899-vvtvr\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") " pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:36:50.970227 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.970189    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-tmp-dir\") pod \"autores-model-f33ec8-kserve-787964f899-vvtvr\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") " pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:36:50.970445 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.970300    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8s8r\" (UniqueName: \"kubernetes.io/projected/02475b45-8fb4-439f-a688-989710da1a50-kube-api-access-q8s8r\") pod \"autores-model-f33ec8-kserve-787964f899-vvtvr\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") " pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:36:50.970445 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.970398    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02475b45-8fb4-439f-a688-989710da1a50-tls-certs\") pod \"autores-model-f33ec8-kserve-787964f899-vvtvr\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") " pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:36:50.970580 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.970514    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-model-cache\") pod \"autores-model-f33ec8-kserve-787964f899-vvtvr\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") " pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:36:50.970580 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.970564    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-tmp-dir\") pod \"autores-model-f33ec8-kserve-787964f899-vvtvr\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") " pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:36:50.970688 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.970584    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-dshm\") pod \"autores-model-f33ec8-kserve-787964f899-vvtvr\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") " pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:36:50.970688 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.970664    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-home\") pod \"autores-model-f33ec8-kserve-787964f899-vvtvr\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") " pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:36:50.970834 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.970811    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-model-cache\") pod \"autores-model-f33ec8-kserve-787964f899-vvtvr\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") " pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:36:50.971238 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.971216    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-home\") pod \"autores-model-f33ec8-kserve-787964f899-vvtvr\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") " pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:36:50.973176 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.973152    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02475b45-8fb4-439f-a688-989710da1a50-tls-certs\") pod \"autores-model-f33ec8-kserve-787964f899-vvtvr\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") " pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:36:50.973643 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.973620    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-dshm\") pod \"autores-model-f33ec8-kserve-787964f899-vvtvr\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") " pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:36:50.978579 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:50.978559    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8s8r\" (UniqueName: \"kubernetes.io/projected/02475b45-8fb4-439f-a688-989710da1a50-kube-api-access-q8s8r\") pod \"autores-model-f33ec8-kserve-787964f899-vvtvr\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") " pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:36:51.108783 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:51.108683    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:36:51.383559 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:36:51.383529    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02475b45_8fb4_439f_a688_989710da1a50.slice/crio-1edef0930c1c463eb82ac5fb051851da82b710a4f446a02a976a0dd3d8961e23 WatchSource:0}: Error finding container 1edef0930c1c463eb82ac5fb051851da82b710a4f446a02a976a0dd3d8961e23: Status 404 returned error can't find the container with id 1edef0930c1c463eb82ac5fb051851da82b710a4f446a02a976a0dd3d8961e23
Sep 15 07:36:51.401183 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:51.401149    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"]
Sep 15 07:36:51.786919 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:51.786885    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr" event={"ID":"02475b45-8fb4-439f-a688-989710da1a50","Type":"ContainerStarted","Data":"74844cab5e5028cb35c412697192b5b2ee783e76e4620babe133807980eed727"}
Sep 15 07:36:51.786919 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:51.786928    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr" event={"ID":"02475b45-8fb4-439f-a688-989710da1a50","Type":"ContainerStarted","Data":"1edef0930c1c463eb82ac5fb051851da82b710a4f446a02a976a0dd3d8961e23"}
Sep 15 07:36:51.805009 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:36:51.804953    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr" podStartSLOduration=1.8049366519999999 podStartE2EDuration="1.804936652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:36:51.803432683 +0000 UTC m=+1116.350002804" watchObservedRunningTime="2026-09-15 07:36:51.804936652 +0000 UTC m=+1116.351506751"
Sep 15 07:37:00.781458 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:00.781421    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7"
Sep 15 07:37:00.781986 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:00.781501    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f"
Sep 15 07:37:01.108941 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:01.108858    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:37:01.108941 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:01.108914    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:37:01.121835 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:01.121802    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:37:01.852624 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:01.852587    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:37:18.185530 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.185492    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7"]
Sep 15 07:37:18.186003 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.185873    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7" podUID="2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca" containerName="payload-processing" containerID="cri-o://64cbc35229cb2ade462c2324e911b21a2e38c48ff40a59316298b5ff81441962" gracePeriod=30
Sep 15 07:37:18.196830 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.196138    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f"]
Sep 15 07:37:18.197325 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.197120    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f" podUID="7309b062-9756-474a-b633-1a9399ad813d" containerName="payload-pre-processing" containerID="cri-o://5f14e6c04782e5115ea1ad0c2a97e1991cfbe5ef3ffbefbd6eb69d83f6ca8502" gracePeriod=30
Sep 15 07:37:18.437878 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.437808    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7"
Sep 15 07:37:18.473608 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.473580    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f"
Sep 15 07:37:18.544524 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.544490    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7309b062-9756-474a-b633-1a9399ad813d-plugins-config-volume\") pod \"7309b062-9756-474a-b633-1a9399ad813d\" (UID: \"7309b062-9756-474a-b633-1a9399ad813d\") "
Sep 15 07:37:18.544696 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.544542    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjz4p\" (UniqueName: \"kubernetes.io/projected/7309b062-9756-474a-b633-1a9399ad813d-kube-api-access-zjz4p\") pod \"7309b062-9756-474a-b633-1a9399ad813d\" (UID: \"7309b062-9756-474a-b633-1a9399ad813d\") "
Sep 15 07:37:18.544696 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.544562    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca-plugins-config-volume\") pod \"2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca\" (UID: \"2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca\") "
Sep 15 07:37:18.544696 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.544616    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plbp4\" (UniqueName: \"kubernetes.io/projected/2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca-kube-api-access-plbp4\") pod \"2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca\" (UID: \"2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca\") "
Sep 15 07:37:18.544991 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.544962    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7309b062-9756-474a-b633-1a9399ad813d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7309b062-9756-474a-b633-1a9399ad813d" (UID: "7309b062-9756-474a-b633-1a9399ad813d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:37:18.545061 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.544962    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca" (UID: "2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:37:18.546832 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.546806    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7309b062-9756-474a-b633-1a9399ad813d-kube-api-access-zjz4p" (OuterVolumeSpecName: "kube-api-access-zjz4p") pod "7309b062-9756-474a-b633-1a9399ad813d" (UID: "7309b062-9756-474a-b633-1a9399ad813d"). InnerVolumeSpecName "kube-api-access-zjz4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:37:18.546924 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.546831    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca-kube-api-access-plbp4" (OuterVolumeSpecName: "kube-api-access-plbp4") pod "2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca" (UID: "2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca"). InnerVolumeSpecName "kube-api-access-plbp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:37:18.645678 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.645643    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7309b062-9756-474a-b633-1a9399ad813d-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:37:18.645678 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.645673    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjz4p\" (UniqueName: \"kubernetes.io/projected/7309b062-9756-474a-b633-1a9399ad813d-kube-api-access-zjz4p\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:37:18.645678 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.645685    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:37:18.645900 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.645693    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plbp4\" (UniqueName: \"kubernetes.io/projected/2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca-kube-api-access-plbp4\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:37:18.939877 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.939841    2571 generic.go:356] "Generic (PLEG): container finished" podID="2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca" containerID="64cbc35229cb2ade462c2324e911b21a2e38c48ff40a59316298b5ff81441962" exitCode=0
Sep 15 07:37:18.940049 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.939887    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7"
Sep 15 07:37:18.940049 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.939932    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7" event={"ID":"2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca","Type":"ContainerDied","Data":"64cbc35229cb2ade462c2324e911b21a2e38c48ff40a59316298b5ff81441962"}
Sep 15 07:37:18.940049 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.939986    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7" event={"ID":"2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca","Type":"ContainerDied","Data":"1b7f19133fb42d8df5d7fc57ad1b19f7ef3cf29ee0d62b6aece8de8f19105a39"}
Sep 15 07:37:18.940049 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.940011    2571 scope.go:117] "RemoveContainer" containerID="64cbc35229cb2ade462c2324e911b21a2e38c48ff40a59316298b5ff81441962"
Sep 15 07:37:18.941441 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.941412    2571 generic.go:356] "Generic (PLEG): container finished" podID="7309b062-9756-474a-b633-1a9399ad813d" containerID="5f14e6c04782e5115ea1ad0c2a97e1991cfbe5ef3ffbefbd6eb69d83f6ca8502" exitCode=0
Sep 15 07:37:18.941522 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.941454    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f" event={"ID":"7309b062-9756-474a-b633-1a9399ad813d","Type":"ContainerDied","Data":"5f14e6c04782e5115ea1ad0c2a97e1991cfbe5ef3ffbefbd6eb69d83f6ca8502"}
Sep 15 07:37:18.941522 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.941462    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f"
Sep 15 07:37:18.941522 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.941492    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f" event={"ID":"7309b062-9756-474a-b633-1a9399ad813d","Type":"ContainerDied","Data":"0c168e8adb6b2da73ec9a29c6d1a6cc147e48dd92f8b5b51cf058852cdadd971"}
Sep 15 07:37:18.949413 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.949393    2571 scope.go:117] "RemoveContainer" containerID="64cbc35229cb2ade462c2324e911b21a2e38c48ff40a59316298b5ff81441962"
Sep 15 07:37:18.949849 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:37:18.949714    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64cbc35229cb2ade462c2324e911b21a2e38c48ff40a59316298b5ff81441962\": container with ID starting with 64cbc35229cb2ade462c2324e911b21a2e38c48ff40a59316298b5ff81441962 not found: ID does not exist" containerID="64cbc35229cb2ade462c2324e911b21a2e38c48ff40a59316298b5ff81441962"
Sep 15 07:37:18.949849 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.949735    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64cbc35229cb2ade462c2324e911b21a2e38c48ff40a59316298b5ff81441962"} err="failed to get container status \"64cbc35229cb2ade462c2324e911b21a2e38c48ff40a59316298b5ff81441962\": rpc error: code = NotFound desc = could not find container \"64cbc35229cb2ade462c2324e911b21a2e38c48ff40a59316298b5ff81441962\": container with ID starting with 64cbc35229cb2ade462c2324e911b21a2e38c48ff40a59316298b5ff81441962 not found: ID does not exist"
Sep 15 07:37:18.949849 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.949750    2571 scope.go:117] "RemoveContainer" containerID="5f14e6c04782e5115ea1ad0c2a97e1991cfbe5ef3ffbefbd6eb69d83f6ca8502"
Sep 15 07:37:18.958150 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.958133    2571 scope.go:117] "RemoveContainer" containerID="5f14e6c04782e5115ea1ad0c2a97e1991cfbe5ef3ffbefbd6eb69d83f6ca8502"
Sep 15 07:37:18.958486 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:37:18.958469    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f14e6c04782e5115ea1ad0c2a97e1991cfbe5ef3ffbefbd6eb69d83f6ca8502\": container with ID starting with 5f14e6c04782e5115ea1ad0c2a97e1991cfbe5ef3ffbefbd6eb69d83f6ca8502 not found: ID does not exist" containerID="5f14e6c04782e5115ea1ad0c2a97e1991cfbe5ef3ffbefbd6eb69d83f6ca8502"
Sep 15 07:37:18.958542 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.958494    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f14e6c04782e5115ea1ad0c2a97e1991cfbe5ef3ffbefbd6eb69d83f6ca8502"} err="failed to get container status \"5f14e6c04782e5115ea1ad0c2a97e1991cfbe5ef3ffbefbd6eb69d83f6ca8502\": rpc error: code = NotFound desc = could not find container \"5f14e6c04782e5115ea1ad0c2a97e1991cfbe5ef3ffbefbd6eb69d83f6ca8502\": container with ID starting with 5f14e6c04782e5115ea1ad0c2a97e1991cfbe5ef3ffbefbd6eb69d83f6ca8502 not found: ID does not exist"
Sep 15 07:37:18.965431 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.965410    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f"]
Sep 15 07:37:18.970032 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.970011    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b63bda62-f9c5bcdf8-4lb5f"]
Sep 15 07:37:18.980856 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.980833    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7"]
Sep 15 07:37:18.987615 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:18.987592    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b63bda62-6bdc67c94c-sntn7"]
Sep 15 07:37:19.909642 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:19.909609    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca" path="/var/lib/kubelet/pods/2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca/volumes"
Sep 15 07:37:19.910035 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:19.909977    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7309b062-9756-474a-b633-1a9399ad813d" path="/var/lib/kubelet/pods/7309b062-9756-474a-b633-1a9399ad813d/volumes"
Sep 15 07:37:23.608575 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:23.608540    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f33ec8-4hvzw"]
Sep 15 07:37:23.609036 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:23.609015    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca" containerName="payload-processing"
Sep 15 07:37:23.609036 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:23.609032    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca" containerName="payload-processing"
Sep 15 07:37:23.609036 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:23.609043    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7309b062-9756-474a-b633-1a9399ad813d" containerName="payload-pre-processing"
Sep 15 07:37:23.609297 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:23.609049    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7309b062-9756-474a-b633-1a9399ad813d" containerName="payload-pre-processing"
Sep 15 07:37:23.609297 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:23.609137    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2de8f3e9-bde1-4d9c-8dd3-0ba86ca3c8ca" containerName="payload-processing"
Sep 15 07:37:23.609297 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:23.609146    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7309b062-9756-474a-b633-1a9399ad813d" containerName="payload-pre-processing"
Sep 15 07:37:23.613515 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:23.613499    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f33ec8-4hvzw"
Sep 15 07:37:23.616841 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:23.616817    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 07:37:23.617073 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:23.617054    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8jpwk\""
Sep 15 07:37:23.619231 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:23.619204    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f33ec8-4hvzw"]
Sep 15 07:37:23.699061 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:23.699029    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e370e4-0125-40d7-b208-46d4200e0f60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-f33ec8-4hvzw\" (UID: \"e7e370e4-0125-40d7-b208-46d4200e0f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f33ec8-4hvzw"
Sep 15 07:37:23.799874 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:23.799837    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e370e4-0125-40d7-b208-46d4200e0f60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-f33ec8-4hvzw\" (UID: \"e7e370e4-0125-40d7-b208-46d4200e0f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f33ec8-4hvzw"
Sep 15 07:37:23.800479 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:23.800457    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e370e4-0125-40d7-b208-46d4200e0f60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-f33ec8-4hvzw\" (UID: \"e7e370e4-0125-40d7-b208-46d4200e0f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f33ec8-4hvzw"
Sep 15 07:37:23.925342 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:23.925242    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f33ec8-4hvzw"
Sep 15 07:37:24.198552 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:24.198523    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f33ec8-4hvzw"]
Sep 15 07:37:24.200465 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:37:24.200433    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e370e4_0125_40d7_b208_46d4200e0f60.slice/crio-6a87b7c092871b43b911198855848da51f68c41e1a1469f7197b8631b151a79d WatchSource:0}: Error finding container 6a87b7c092871b43b911198855848da51f68c41e1a1469f7197b8631b151a79d: Status 404 returned error can't find the container with id 6a87b7c092871b43b911198855848da51f68c41e1a1469f7197b8631b151a79d
Sep 15 07:37:24.202480 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:24.202461    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 07:37:24.985581 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:24.985547    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f33ec8-4hvzw" event={"ID":"e7e370e4-0125-40d7-b208-46d4200e0f60","Type":"ContainerStarted","Data":"6a87b7c092871b43b911198855848da51f68c41e1a1469f7197b8631b151a79d"}
Sep 15 07:37:29.244030 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:29.243998    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq"]
Sep 15 07:37:29.248720 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:29.248702    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq"
Sep 15 07:37:29.251171 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:29.251149    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-67d9aa69-dockercfg-4hblc\""
Sep 15 07:37:29.251305 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:29.251201    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-67d9aa69\""
Sep 15 07:37:29.257555 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:29.257407    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq"]
Sep 15 07:37:29.354706 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:29.354670    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777ea164-3203-4451-b71b-6a4488c71079-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq\" (UID: \"777ea164-3203-4451-b71b-6a4488c71079\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq"
Sep 15 07:37:29.354882 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:29.354783    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l254\" (UniqueName: \"kubernetes.io/projected/777ea164-3203-4451-b71b-6a4488c71079-kube-api-access-5l254\") pod \"payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq\" (UID: \"777ea164-3203-4451-b71b-6a4488c71079\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq"
Sep 15 07:37:29.456324 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:29.456289    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l254\" (UniqueName: \"kubernetes.io/projected/777ea164-3203-4451-b71b-6a4488c71079-kube-api-access-5l254\") pod \"payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq\" (UID: \"777ea164-3203-4451-b71b-6a4488c71079\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq"
Sep 15 07:37:29.456494 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:29.456388    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777ea164-3203-4451-b71b-6a4488c71079-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq\" (UID: \"777ea164-3203-4451-b71b-6a4488c71079\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq"
Sep 15 07:37:29.457018 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:29.457000    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777ea164-3203-4451-b71b-6a4488c71079-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq\" (UID: \"777ea164-3203-4451-b71b-6a4488c71079\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq"
Sep 15 07:37:29.465276 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:29.465226    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l254\" (UniqueName: \"kubernetes.io/projected/777ea164-3203-4451-b71b-6a4488c71079-kube-api-access-5l254\") pod \"payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq\" (UID: \"777ea164-3203-4451-b71b-6a4488c71079\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq"
Sep 15 07:37:29.561524 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:29.561438    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq"
Sep 15 07:37:29.831400 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:29.831368    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq"]
Sep 15 07:37:29.833611 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:37:29.833588    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777ea164_3203_4451_b71b_6a4488c71079.slice/crio-f6c17553358717df38ca50fda1e534a6f0c4536f4085fdc19d7de3e08688ccba WatchSource:0}: Error finding container f6c17553358717df38ca50fda1e534a6f0c4536f4085fdc19d7de3e08688ccba: Status 404 returned error can't find the container with id f6c17553358717df38ca50fda1e534a6f0c4536f4085fdc19d7de3e08688ccba
Sep 15 07:37:30.013842 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:30.013796    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq" event={"ID":"777ea164-3203-4451-b71b-6a4488c71079","Type":"ContainerStarted","Data":"1d737edd2afa73dcba544c43b1998c0fddb31f9601542742c20756ab5e123a3e"}
Sep 15 07:37:30.014014 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:30.013865    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq" event={"ID":"777ea164-3203-4451-b71b-6a4488c71079","Type":"ContainerStarted","Data":"f6c17553358717df38ca50fda1e534a6f0c4536f4085fdc19d7de3e08688ccba"}
Sep 15 07:37:30.014158 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:30.014017    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq"
Sep 15 07:37:30.034228 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:30.034180    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq" podStartSLOduration=1.0341658950000001 podStartE2EDuration="1.034165895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:37:30.031184629 +0000 UTC m=+1154.577754753" watchObservedRunningTime="2026-09-15 07:37:30.034165895 +0000 UTC m=+1154.580735994"
Sep 15 07:37:35.046553 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:35.046466    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f33ec8-4hvzw" event={"ID":"e7e370e4-0125-40d7-b208-46d4200e0f60","Type":"ContainerStarted","Data":"7cfb37b15f2e8fe4c475155304af524018bc6a8a6d6fecdbd4742be6f5e01509"}
Sep 15 07:37:35.078817 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:35.078768    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f33ec8-4hvzw" podStartSLOduration=1.8341877229999999 podStartE2EDuration="12.078753287s" totalImagesPullingTime="10.244565564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:37:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:37:35.077632844 +0000 UTC m=+1159.624202943" watchObservedRunningTime="2026-09-15 07:37:35.078753287 +0000 UTC m=+1159.625323386"
Sep 15 07:37:36.053636 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:36.053602    2571 generic.go:356] "Generic (PLEG): container finished" podID="e7e370e4-0125-40d7-b208-46d4200e0f60" containerID="7cfb37b15f2e8fe4c475155304af524018bc6a8a6d6fecdbd4742be6f5e01509" exitCode=0
Sep 15 07:37:36.054047 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:36.053688    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f33ec8-4hvzw" event={"ID":"e7e370e4-0125-40d7-b208-46d4200e0f60","Type":"ContainerDied","Data":"7cfb37b15f2e8fe4c475155304af524018bc6a8a6d6fecdbd4742be6f5e01509"}
Sep 15 07:37:37.193230 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:37.193206    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f33ec8-4hvzw"
Sep 15 07:37:37.339602 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:37.339513    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e370e4-0125-40d7-b208-46d4200e0f60-maas-api-service-ca\") pod \"e7e370e4-0125-40d7-b208-46d4200e0f60\" (UID: \"e7e370e4-0125-40d7-b208-46d4200e0f60\") "
Sep 15 07:37:37.339879 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:37.339855    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e370e4-0125-40d7-b208-46d4200e0f60-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e7e370e4-0125-40d7-b208-46d4200e0f60" (UID: "e7e370e4-0125-40d7-b208-46d4200e0f60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:37:37.440692 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:37.440660    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e370e4-0125-40d7-b208-46d4200e0f60-maas-api-service-ca\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:37:38.066440 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:38.066410    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f33ec8-4hvzw"
Sep 15 07:37:38.066608 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:38.066450    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f33ec8-4hvzw" event={"ID":"e7e370e4-0125-40d7-b208-46d4200e0f60","Type":"ContainerDied","Data":"6a87b7c092871b43b911198855848da51f68c41e1a1469f7197b8631b151a79d"}
Sep 15 07:37:38.066608 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:38.066500    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a87b7c092871b43b911198855848da51f68c41e1a1469f7197b8631b151a79d"
Sep 15 07:37:41.023418 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:41.023385    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq"
Sep 15 07:37:44.428645 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:44.428605    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx"]
Sep 15 07:37:44.429129 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:44.428856    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx" podUID="a72daa9a-2303-445d-9aad-717a590380e9" containerName="payload-pre-processing" containerID="cri-o://f1844b0b8c899b50c110d2d8d3ebb48ba9eb5b09631aa91091d32627fe1d8df3" gracePeriod=30
Sep 15 07:37:44.589924 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:44.589878    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f33ec8-4hvzw"]
Sep 15 07:37:44.592070 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:44.592042    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f33ec8-4hvzw"]
Sep 15 07:37:44.689508 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:44.689439    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx"
Sep 15 07:37:44.722674 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:44.722643    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a72daa9a-2303-445d-9aad-717a590380e9-plugins-config-volume\") pod \"a72daa9a-2303-445d-9aad-717a590380e9\" (UID: \"a72daa9a-2303-445d-9aad-717a590380e9\") "
Sep 15 07:37:44.722852 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:44.722743    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxh85\" (UniqueName: \"kubernetes.io/projected/a72daa9a-2303-445d-9aad-717a590380e9-kube-api-access-hxh85\") pod \"a72daa9a-2303-445d-9aad-717a590380e9\" (UID: \"a72daa9a-2303-445d-9aad-717a590380e9\") "
Sep 15 07:37:44.723007 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:44.722984    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a72daa9a-2303-445d-9aad-717a590380e9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a72daa9a-2303-445d-9aad-717a590380e9" (UID: "a72daa9a-2303-445d-9aad-717a590380e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:37:44.724849 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:44.724825    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72daa9a-2303-445d-9aad-717a590380e9-kube-api-access-hxh85" (OuterVolumeSpecName: "kube-api-access-hxh85") pod "a72daa9a-2303-445d-9aad-717a590380e9" (UID: "a72daa9a-2303-445d-9aad-717a590380e9"). InnerVolumeSpecName "kube-api-access-hxh85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:37:44.824029 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:44.823988    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxh85\" (UniqueName: \"kubernetes.io/projected/a72daa9a-2303-445d-9aad-717a590380e9-kube-api-access-hxh85\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:37:44.824029 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:44.824024    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a72daa9a-2303-445d-9aad-717a590380e9-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:37:45.105022 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:45.104981    2571 generic.go:356] "Generic (PLEG): container finished" podID="a72daa9a-2303-445d-9aad-717a590380e9" containerID="f1844b0b8c899b50c110d2d8d3ebb48ba9eb5b09631aa91091d32627fe1d8df3" exitCode=0
Sep 15 07:37:45.105177 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:45.105028    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx"
Sep 15 07:37:45.105177 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:45.105063    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx" event={"ID":"a72daa9a-2303-445d-9aad-717a590380e9","Type":"ContainerDied","Data":"f1844b0b8c899b50c110d2d8d3ebb48ba9eb5b09631aa91091d32627fe1d8df3"}
Sep 15 07:37:45.105177 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:45.105120    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx" event={"ID":"a72daa9a-2303-445d-9aad-717a590380e9","Type":"ContainerDied","Data":"410e05ec278ecc1cce7cad36843b0fa1ab05cea2ced2c88cbb875a0dbd237e17"}
Sep 15 07:37:45.105177 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:45.105143    2571 scope.go:117] "RemoveContainer" containerID="f1844b0b8c899b50c110d2d8d3ebb48ba9eb5b09631aa91091d32627fe1d8df3"
Sep 15 07:37:45.114382 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:45.114363    2571 scope.go:117] "RemoveContainer" containerID="f1844b0b8c899b50c110d2d8d3ebb48ba9eb5b09631aa91091d32627fe1d8df3"
Sep 15 07:37:45.114648 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:37:45.114630    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1844b0b8c899b50c110d2d8d3ebb48ba9eb5b09631aa91091d32627fe1d8df3\": container with ID starting with f1844b0b8c899b50c110d2d8d3ebb48ba9eb5b09631aa91091d32627fe1d8df3 not found: ID does not exist" containerID="f1844b0b8c899b50c110d2d8d3ebb48ba9eb5b09631aa91091d32627fe1d8df3"
Sep 15 07:37:45.114697 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:45.114657    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1844b0b8c899b50c110d2d8d3ebb48ba9eb5b09631aa91091d32627fe1d8df3"} err="failed to get container status \"f1844b0b8c899b50c110d2d8d3ebb48ba9eb5b09631aa91091d32627fe1d8df3\": rpc error: code = NotFound desc = could not find container \"f1844b0b8c899b50c110d2d8d3ebb48ba9eb5b09631aa91091d32627fe1d8df3\": container with ID starting with f1844b0b8c899b50c110d2d8d3ebb48ba9eb5b09631aa91091d32627fe1d8df3 not found: ID does not exist"
Sep 15 07:37:45.127122 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:45.127099    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx"]
Sep 15 07:37:45.129095 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:45.129076    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f33ec8-768d8888bb-bbrbx"]
Sep 15 07:37:45.910306 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:45.910272    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a72daa9a-2303-445d-9aad-717a590380e9" path="/var/lib/kubelet/pods/a72daa9a-2303-445d-9aad-717a590380e9/volumes"
Sep 15 07:37:45.910672 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:45.910608    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7e370e4-0125-40d7-b208-46d4200e0f60" path="/var/lib/kubelet/pods/e7e370e4-0125-40d7-b208-46d4200e0f60/volumes"
Sep 15 07:37:49.323970 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.323927    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"]
Sep 15 07:37:49.324719 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.324695    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7e370e4-0125-40d7-b208-46d4200e0f60" containerName="revoke-keys"
Sep 15 07:37:49.324800 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.324722    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7e370e4-0125-40d7-b208-46d4200e0f60" containerName="revoke-keys"
Sep 15 07:37:49.324800 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.324735    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a72daa9a-2303-445d-9aad-717a590380e9" containerName="payload-pre-processing"
Sep 15 07:37:49.324800 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.324745    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a72daa9a-2303-445d-9aad-717a590380e9" containerName="payload-pre-processing"
Sep 15 07:37:49.324896 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.324879    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a72daa9a-2303-445d-9aad-717a590380e9" containerName="payload-pre-processing"
Sep 15 07:37:49.324928 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.324896    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7e370e4-0125-40d7-b208-46d4200e0f60" containerName="revoke-keys"
Sep 15 07:37:49.329675 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.329652    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:37:49.332337 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.332313    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ee0096\"/\"openshift-service-ca.crt\""
Sep 15 07:37:49.333360 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.333335    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ee0096\"/\"default-dockercfg-tp5mx\""
Sep 15 07:37:49.333731 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.333407    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ee0096\"/\"kube-root-ca.crt\""
Sep 15 07:37:49.333731 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.333476    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ee0096\"/\"test-model-ee0096-kserve-self-signed-certs\""
Sep 15 07:37:49.337356 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.337334    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"]
Sep 15 07:37:49.366219 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.366186    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-model-cache\") pod \"test-model-ee0096-kserve-77f5bb9fd7-c5jpm\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") " pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:37:49.366219 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.366219    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-tmp-dir\") pod \"test-model-ee0096-kserve-77f5bb9fd7-c5jpm\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") " pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:37:49.366444 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.366264    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h95rf\" (UniqueName: \"kubernetes.io/projected/d3f34c4a-b956-4c19-aaee-f85fdef80b99-kube-api-access-h95rf\") pod \"test-model-ee0096-kserve-77f5bb9fd7-c5jpm\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") " pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:37:49.366444 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.366375    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-home\") pod \"test-model-ee0096-kserve-77f5bb9fd7-c5jpm\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") " pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:37:49.366444 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.366405    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f34c4a-b956-4c19-aaee-f85fdef80b99-tls-certs\") pod \"test-model-ee0096-kserve-77f5bb9fd7-c5jpm\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") " pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:37:49.366541 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.366448    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-dshm\") pod \"test-model-ee0096-kserve-77f5bb9fd7-c5jpm\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") " pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:37:49.467392 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.467355    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-home\") pod \"test-model-ee0096-kserve-77f5bb9fd7-c5jpm\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") " pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:37:49.467392 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.467396    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f34c4a-b956-4c19-aaee-f85fdef80b99-tls-certs\") pod \"test-model-ee0096-kserve-77f5bb9fd7-c5jpm\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") " pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:37:49.467603 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.467435    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-dshm\") pod \"test-model-ee0096-kserve-77f5bb9fd7-c5jpm\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") " pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:37:49.467603 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.467501    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-model-cache\") pod \"test-model-ee0096-kserve-77f5bb9fd7-c5jpm\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") " pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:37:49.467689 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.467622    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-tmp-dir\") pod \"test-model-ee0096-kserve-77f5bb9fd7-c5jpm\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") " pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:37:49.467743 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.467702    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h95rf\" (UniqueName: \"kubernetes.io/projected/d3f34c4a-b956-4c19-aaee-f85fdef80b99-kube-api-access-h95rf\") pod \"test-model-ee0096-kserve-77f5bb9fd7-c5jpm\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") " pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:37:49.467798 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.467762    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-home\") pod \"test-model-ee0096-kserve-77f5bb9fd7-c5jpm\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") " pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:37:49.467862 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.467817    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-model-cache\") pod \"test-model-ee0096-kserve-77f5bb9fd7-c5jpm\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") " pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:37:49.467992 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.467972    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-tmp-dir\") pod \"test-model-ee0096-kserve-77f5bb9fd7-c5jpm\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") " pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:37:49.469784 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.469763    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-dshm\") pod \"test-model-ee0096-kserve-77f5bb9fd7-c5jpm\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") " pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:37:49.469952 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.469935    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f34c4a-b956-4c19-aaee-f85fdef80b99-tls-certs\") pod \"test-model-ee0096-kserve-77f5bb9fd7-c5jpm\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") " pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:37:49.475938 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.475919    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h95rf\" (UniqueName: \"kubernetes.io/projected/d3f34c4a-b956-4c19-aaee-f85fdef80b99-kube-api-access-h95rf\") pod \"test-model-ee0096-kserve-77f5bb9fd7-c5jpm\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") " pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:37:49.639663 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.639566    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:37:49.912966 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.912936    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"]
Sep 15 07:37:49.914649 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:37:49.914609    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3f34c4a_b956_4c19_aaee_f85fdef80b99.slice/crio-4b7cb2708bea3fa875228ff9f563827af582a2b74f4c048e34c91babf7a85a93 WatchSource:0}: Error finding container 4b7cb2708bea3fa875228ff9f563827af582a2b74f4c048e34c91babf7a85a93: Status 404 returned error can't find the container with id 4b7cb2708bea3fa875228ff9f563827af582a2b74f4c048e34c91babf7a85a93
Sep 15 07:37:49.938501 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:49.938473    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"]
Sep 15 07:37:50.134817 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:50.134777    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm" event={"ID":"d3f34c4a-b956-4c19-aaee-f85fdef80b99","Type":"ContainerStarted","Data":"15a62fa8ff3e8e8f9eac5b59e1ec6563d9b9e3bebf8d279ad8aa8e336e50fb00"}
Sep 15 07:37:50.134817 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:50.134822    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm" event={"ID":"d3f34c4a-b956-4c19-aaee-f85fdef80b99","Type":"ContainerStarted","Data":"4b7cb2708bea3fa875228ff9f563827af582a2b74f4c048e34c91babf7a85a93"}
Sep 15 07:37:50.155943 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:50.155890    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm" podStartSLOduration=1.155874129 podStartE2EDuration="1.155874129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:37:50.154122841 +0000 UTC m=+1174.700692941" watchObservedRunningTime="2026-09-15 07:37:50.155874129 +0000 UTC m=+1174.702444227"
Sep 15 07:37:52.772356 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:52.772323    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67d9aa69-mv96v"]
Sep 15 07:37:52.775109 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:52.775089    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67d9aa69-mv96v"
Sep 15 07:37:52.777314 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:52.777288    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 07:37:52.777314 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:52.777310    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8jpwk\""
Sep 15 07:37:52.782446 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:52.782422    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67d9aa69-mv96v"]
Sep 15 07:37:52.801157 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:52.801124    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/108aa838-0899-4712-8200-664ae1d28675-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67d9aa69-mv96v\" (UID: \"108aa838-0899-4712-8200-664ae1d28675\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67d9aa69-mv96v"
Sep 15 07:37:52.902303 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:52.902240    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/108aa838-0899-4712-8200-664ae1d28675-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67d9aa69-mv96v\" (UID: \"108aa838-0899-4712-8200-664ae1d28675\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67d9aa69-mv96v"
Sep 15 07:37:52.902876 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:52.902854    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/108aa838-0899-4712-8200-664ae1d28675-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67d9aa69-mv96v\" (UID: \"108aa838-0899-4712-8200-664ae1d28675\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67d9aa69-mv96v"
Sep 15 07:37:53.088150 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:53.088052    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67d9aa69-mv96v"
Sep 15 07:37:53.368991 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:53.368964    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67d9aa69-mv96v"]
Sep 15 07:37:53.370978 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:37:53.370952    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108aa838_0899_4712_8200_664ae1d28675.slice/crio-22dfd5035aab392c9bb74eebe9054646490190d42f042ea023a2dac49558a5ea WatchSource:0}: Error finding container 22dfd5035aab392c9bb74eebe9054646490190d42f042ea023a2dac49558a5ea: Status 404 returned error can't find the container with id 22dfd5035aab392c9bb74eebe9054646490190d42f042ea023a2dac49558a5ea
Sep 15 07:37:54.179646 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:54.179607    2571 generic.go:356] "Generic (PLEG): container finished" podID="108aa838-0899-4712-8200-664ae1d28675" containerID="6a395779463aa62e5ebce2002e6d831d9cc741160f1b45f48b6440cf56911bb0" exitCode=0
Sep 15 07:37:54.180016 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:54.179696    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67d9aa69-mv96v" event={"ID":"108aa838-0899-4712-8200-664ae1d28675","Type":"ContainerDied","Data":"6a395779463aa62e5ebce2002e6d831d9cc741160f1b45f48b6440cf56911bb0"}
Sep 15 07:37:54.180016 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:54.179743    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67d9aa69-mv96v" event={"ID":"108aa838-0899-4712-8200-664ae1d28675","Type":"ContainerStarted","Data":"22dfd5035aab392c9bb74eebe9054646490190d42f042ea023a2dac49558a5ea"}
Sep 15 07:37:55.326108 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:55.326079    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67d9aa69-mv96v"
Sep 15 07:37:55.426103 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:55.426061    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/108aa838-0899-4712-8200-664ae1d28675-maas-api-service-ca\") pod \"108aa838-0899-4712-8200-664ae1d28675\" (UID: \"108aa838-0899-4712-8200-664ae1d28675\") "
Sep 15 07:37:55.426447 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:55.426423    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/108aa838-0899-4712-8200-664ae1d28675-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "108aa838-0899-4712-8200-664ae1d28675" (UID: "108aa838-0899-4712-8200-664ae1d28675"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:37:55.527772 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:55.527732    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/108aa838-0899-4712-8200-664ae1d28675-maas-api-service-ca\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:37:56.192505 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:56.192421    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67d9aa69-mv96v"
Sep 15 07:37:56.192505 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:56.192450    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67d9aa69-mv96v" event={"ID":"108aa838-0899-4712-8200-664ae1d28675","Type":"ContainerDied","Data":"22dfd5035aab392c9bb74eebe9054646490190d42f042ea023a2dac49558a5ea"}
Sep 15 07:37:56.192686 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:56.192506    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22dfd5035aab392c9bb74eebe9054646490190d42f042ea023a2dac49558a5ea"
Sep 15 07:37:59.639733 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:59.639684    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:37:59.640101 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:59.639753    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:37:59.652231 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:37:59.652204    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:38:00.232187 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:00.232158    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:38:04.959791 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:04.959716    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr" podUID="02475b45-8fb4-439f-a688-989710da1a50" containerName="main" containerID="cri-o://74844cab5e5028cb35c412697192b5b2ee783e76e4620babe133807980eed727" gracePeriod=45
Sep 15 07:38:05.225848 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.225771    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:38:05.252699 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.252659    2571 generic.go:356] "Generic (PLEG): container finished" podID="02475b45-8fb4-439f-a688-989710da1a50" containerID="74844cab5e5028cb35c412697192b5b2ee783e76e4620babe133807980eed727" exitCode=0
Sep 15 07:38:05.252919 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.252803    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr" event={"ID":"02475b45-8fb4-439f-a688-989710da1a50","Type":"ContainerDied","Data":"74844cab5e5028cb35c412697192b5b2ee783e76e4620babe133807980eed727"}
Sep 15 07:38:05.252919 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.252848    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr" event={"ID":"02475b45-8fb4-439f-a688-989710da1a50","Type":"ContainerDied","Data":"1edef0930c1c463eb82ac5fb051851da82b710a4f446a02a976a0dd3d8961e23"}
Sep 15 07:38:05.252919 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.252873    2571 scope.go:117] "RemoveContainer" containerID="74844cab5e5028cb35c412697192b5b2ee783e76e4620babe133807980eed727"
Sep 15 07:38:05.252919 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.252889    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"
Sep 15 07:38:05.262145 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.262117    2571 scope.go:117] "RemoveContainer" containerID="74844cab5e5028cb35c412697192b5b2ee783e76e4620babe133807980eed727"
Sep 15 07:38:05.262490 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:38:05.262467    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74844cab5e5028cb35c412697192b5b2ee783e76e4620babe133807980eed727\": container with ID starting with 74844cab5e5028cb35c412697192b5b2ee783e76e4620babe133807980eed727 not found: ID does not exist" containerID="74844cab5e5028cb35c412697192b5b2ee783e76e4620babe133807980eed727"
Sep 15 07:38:05.262610 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.262498    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74844cab5e5028cb35c412697192b5b2ee783e76e4620babe133807980eed727"} err="failed to get container status \"74844cab5e5028cb35c412697192b5b2ee783e76e4620babe133807980eed727\": rpc error: code = NotFound desc = could not find container \"74844cab5e5028cb35c412697192b5b2ee783e76e4620babe133807980eed727\": container with ID starting with 74844cab5e5028cb35c412697192b5b2ee783e76e4620babe133807980eed727 not found: ID does not exist"
Sep 15 07:38:05.327179 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.327147    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-home\") pod \"02475b45-8fb4-439f-a688-989710da1a50\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") "
Sep 15 07:38:05.327398 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.327214    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-model-cache\") pod \"02475b45-8fb4-439f-a688-989710da1a50\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") "
Sep 15 07:38:05.327398 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.327241    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-dshm\") pod \"02475b45-8fb4-439f-a688-989710da1a50\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") "
Sep 15 07:38:05.327398 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.327298    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8s8r\" (UniqueName: \"kubernetes.io/projected/02475b45-8fb4-439f-a688-989710da1a50-kube-api-access-q8s8r\") pod \"02475b45-8fb4-439f-a688-989710da1a50\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") "
Sep 15 07:38:05.327398 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.327346    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-tmp-dir\") pod \"02475b45-8fb4-439f-a688-989710da1a50\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") "
Sep 15 07:38:05.327398 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.327373    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02475b45-8fb4-439f-a688-989710da1a50-tls-certs\") pod \"02475b45-8fb4-439f-a688-989710da1a50\" (UID: \"02475b45-8fb4-439f-a688-989710da1a50\") "
Sep 15 07:38:05.327686 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.327429    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-home" (OuterVolumeSpecName: "home") pod "02475b45-8fb4-439f-a688-989710da1a50" (UID: "02475b45-8fb4-439f-a688-989710da1a50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:38:05.327686 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.327463    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-model-cache" (OuterVolumeSpecName: "model-cache") pod "02475b45-8fb4-439f-a688-989710da1a50" (UID: "02475b45-8fb4-439f-a688-989710da1a50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:38:05.327686 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.327592    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "02475b45-8fb4-439f-a688-989710da1a50" (UID: "02475b45-8fb4-439f-a688-989710da1a50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:38:05.327859 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.327764    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-home\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:38:05.327859 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.327782    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-model-cache\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:38:05.327859 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.327796    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-tmp-dir\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:38:05.329532 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.329506    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-dshm" (OuterVolumeSpecName: "dshm") pod "02475b45-8fb4-439f-a688-989710da1a50" (UID: "02475b45-8fb4-439f-a688-989710da1a50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:38:05.329928 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.329902    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02475b45-8fb4-439f-a688-989710da1a50-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "02475b45-8fb4-439f-a688-989710da1a50" (UID: "02475b45-8fb4-439f-a688-989710da1a50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:38:05.330025 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.329920    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02475b45-8fb4-439f-a688-989710da1a50-kube-api-access-q8s8r" (OuterVolumeSpecName: "kube-api-access-q8s8r") pod "02475b45-8fb4-439f-a688-989710da1a50" (UID: "02475b45-8fb4-439f-a688-989710da1a50"). InnerVolumeSpecName "kube-api-access-q8s8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:38:05.391122 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.391082    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq"]
Sep 15 07:38:05.391350 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.391326    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq" podUID="777ea164-3203-4451-b71b-6a4488c71079" containerName="payload-pre-processing" containerID="cri-o://1d737edd2afa73dcba544c43b1998c0fddb31f9601542742c20756ab5e123a3e" gracePeriod=30
Sep 15 07:38:05.428482 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.428451    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02475b45-8fb4-439f-a688-989710da1a50-dshm\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:38:05.428482 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.428479    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8s8r\" (UniqueName: \"kubernetes.io/projected/02475b45-8fb4-439f-a688-989710da1a50-kube-api-access-q8s8r\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:38:05.428651 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.428491    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02475b45-8fb4-439f-a688-989710da1a50-tls-certs\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:38:05.565204 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.565175    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67d9aa69-mv96v"]
Sep 15 07:38:05.569195 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.569171    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67d9aa69-mv96v"]
Sep 15 07:38:05.575894 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.575868    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"]
Sep 15 07:38:05.582638 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.582480    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-f33ec8/autores-model-f33ec8-kserve-787964f899-vvtvr"]
Sep 15 07:38:05.635407 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.635381    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq"
Sep 15 07:38:05.731765 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.731733    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777ea164-3203-4451-b71b-6a4488c71079-plugins-config-volume\") pod \"777ea164-3203-4451-b71b-6a4488c71079\" (UID: \"777ea164-3203-4451-b71b-6a4488c71079\") "
Sep 15 07:38:05.731929 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.731788    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5l254\" (UniqueName: \"kubernetes.io/projected/777ea164-3203-4451-b71b-6a4488c71079-kube-api-access-5l254\") pod \"777ea164-3203-4451-b71b-6a4488c71079\" (UID: \"777ea164-3203-4451-b71b-6a4488c71079\") "
Sep 15 07:38:05.732174 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.732152    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777ea164-3203-4451-b71b-6a4488c71079-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "777ea164-3203-4451-b71b-6a4488c71079" (UID: "777ea164-3203-4451-b71b-6a4488c71079"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:38:05.733769 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.733747    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777ea164-3203-4451-b71b-6a4488c71079-kube-api-access-5l254" (OuterVolumeSpecName: "kube-api-access-5l254") pod "777ea164-3203-4451-b71b-6a4488c71079" (UID: "777ea164-3203-4451-b71b-6a4488c71079"). InnerVolumeSpecName "kube-api-access-5l254". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:38:05.833291 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.833172    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l254\" (UniqueName: \"kubernetes.io/projected/777ea164-3203-4451-b71b-6a4488c71079-kube-api-access-5l254\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:38:05.833291 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.833207    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777ea164-3203-4451-b71b-6a4488c71079-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:38:05.910406 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.910372    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02475b45-8fb4-439f-a688-989710da1a50" path="/var/lib/kubelet/pods/02475b45-8fb4-439f-a688-989710da1a50/volumes"
Sep 15 07:38:05.910744 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:05.910732    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="108aa838-0899-4712-8200-664ae1d28675" path="/var/lib/kubelet/pods/108aa838-0899-4712-8200-664ae1d28675/volumes"
Sep 15 07:38:06.260510 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:06.260474    2571 generic.go:356] "Generic (PLEG): container finished" podID="777ea164-3203-4451-b71b-6a4488c71079" containerID="1d737edd2afa73dcba544c43b1998c0fddb31f9601542742c20756ab5e123a3e" exitCode=0
Sep 15 07:38:06.260952 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:06.260520    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq"
Sep 15 07:38:06.260952 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:06.260522    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq" event={"ID":"777ea164-3203-4451-b71b-6a4488c71079","Type":"ContainerDied","Data":"1d737edd2afa73dcba544c43b1998c0fddb31f9601542742c20756ab5e123a3e"}
Sep 15 07:38:06.260952 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:06.260651    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq" event={"ID":"777ea164-3203-4451-b71b-6a4488c71079","Type":"ContainerDied","Data":"f6c17553358717df38ca50fda1e534a6f0c4536f4085fdc19d7de3e08688ccba"}
Sep 15 07:38:06.260952 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:06.260680    2571 scope.go:117] "RemoveContainer" containerID="1d737edd2afa73dcba544c43b1998c0fddb31f9601542742c20756ab5e123a3e"
Sep 15 07:38:06.269842 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:06.269819    2571 scope.go:117] "RemoveContainer" containerID="1d737edd2afa73dcba544c43b1998c0fddb31f9601542742c20756ab5e123a3e"
Sep 15 07:38:06.270129 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:38:06.270108    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d737edd2afa73dcba544c43b1998c0fddb31f9601542742c20756ab5e123a3e\": container with ID starting with 1d737edd2afa73dcba544c43b1998c0fddb31f9601542742c20756ab5e123a3e not found: ID does not exist" containerID="1d737edd2afa73dcba544c43b1998c0fddb31f9601542742c20756ab5e123a3e"
Sep 15 07:38:06.270177 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:06.270139    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d737edd2afa73dcba544c43b1998c0fddb31f9601542742c20756ab5e123a3e"} err="failed to get container status \"1d737edd2afa73dcba544c43b1998c0fddb31f9601542742c20756ab5e123a3e\": rpc error: code = NotFound desc = could not find container \"1d737edd2afa73dcba544c43b1998c0fddb31f9601542742c20756ab5e123a3e\": container with ID starting with 1d737edd2afa73dcba544c43b1998c0fddb31f9601542742c20756ab5e123a3e not found: ID does not exist"
Sep 15 07:38:06.278673 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:06.278644    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq"]
Sep 15 07:38:06.282406 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:06.282383    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-67d9aa69-6575b9d498-vsktq"]
Sep 15 07:38:07.910232 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:38:07.910192    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="777ea164-3203-4451-b71b-6a4488c71079" path="/var/lib/kubelet/pods/777ea164-3203-4451-b71b-6a4488c71079/volumes"
Sep 15 07:39:28.608593 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.608515    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"]
Sep 15 07:39:28.611145 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.608993    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="777ea164-3203-4451-b71b-6a4488c71079" containerName="payload-pre-processing"
Sep 15 07:39:28.611145 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.609008    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="777ea164-3203-4451-b71b-6a4488c71079" containerName="payload-pre-processing"
Sep 15 07:39:28.611145 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.609017    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="108aa838-0899-4712-8200-664ae1d28675" containerName="revoke-keys"
Sep 15 07:39:28.611145 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.609024    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="108aa838-0899-4712-8200-664ae1d28675" containerName="revoke-keys"
Sep 15 07:39:28.611145 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.609033    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02475b45-8fb4-439f-a688-989710da1a50" containerName="main"
Sep 15 07:39:28.611145 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.609039    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="02475b45-8fb4-439f-a688-989710da1a50" containerName="main"
Sep 15 07:39:28.611145 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.609113    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="777ea164-3203-4451-b71b-6a4488c71079" containerName="payload-pre-processing"
Sep 15 07:39:28.611145 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.609123    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="02475b45-8fb4-439f-a688-989710da1a50" containerName="main"
Sep 15 07:39:28.611145 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.609129    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="108aa838-0899-4712-8200-664ae1d28675" containerName="revoke-keys"
Sep 15 07:39:28.612249 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.612234    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:28.614561 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.614540    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fcd453\"/\"openshift-service-ca.crt\""
Sep 15 07:39:28.615665 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.615648    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fcd453\"/\"kube-root-ca.crt\""
Sep 15 07:39:28.615762 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.615663    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fcd453\"/\"default-dockercfg-f4fgk\""
Sep 15 07:39:28.615762 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.615672    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fcd453\"/\"test-model-fcd453-kserve-self-signed-certs\""
Sep 15 07:39:28.623166 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.623144    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"]
Sep 15 07:39:28.752792 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.752759    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-model-cache\") pod \"test-model-fcd453-kserve-c78759c9-96pct\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") " pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:28.752982 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.752828    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4e29f82-02fc-42ba-a375-f8d9bc88123c-tls-certs\") pod \"test-model-fcd453-kserve-c78759c9-96pct\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") " pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:28.752982 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.752866    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-tmp-dir\") pod \"test-model-fcd453-kserve-c78759c9-96pct\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") " pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:28.752982 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.752933    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-home\") pod \"test-model-fcd453-kserve-c78759c9-96pct\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") " pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:28.752982 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.752968    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-dshm\") pod \"test-model-fcd453-kserve-c78759c9-96pct\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") " pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:28.753180 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.753039    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5lxp\" (UniqueName: \"kubernetes.io/projected/c4e29f82-02fc-42ba-a375-f8d9bc88123c-kube-api-access-l5lxp\") pod \"test-model-fcd453-kserve-c78759c9-96pct\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") " pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:28.853986 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.853953    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4e29f82-02fc-42ba-a375-f8d9bc88123c-tls-certs\") pod \"test-model-fcd453-kserve-c78759c9-96pct\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") " pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:28.853986 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.853988    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-tmp-dir\") pod \"test-model-fcd453-kserve-c78759c9-96pct\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") " pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:28.854228 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.854097    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-home\") pod \"test-model-fcd453-kserve-c78759c9-96pct\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") " pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:28.854228 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.854117    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-dshm\") pod \"test-model-fcd453-kserve-c78759c9-96pct\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") " pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:28.854228 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.854149    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5lxp\" (UniqueName: \"kubernetes.io/projected/c4e29f82-02fc-42ba-a375-f8d9bc88123c-kube-api-access-l5lxp\") pod \"test-model-fcd453-kserve-c78759c9-96pct\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") " pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:28.854228 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.854177    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-model-cache\") pod \"test-model-fcd453-kserve-c78759c9-96pct\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") " pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:28.854619 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.854586    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-model-cache\") pod \"test-model-fcd453-kserve-c78759c9-96pct\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") " pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:28.854619 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.854599    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-home\") pod \"test-model-fcd453-kserve-c78759c9-96pct\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") " pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:28.854766 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.854620    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-tmp-dir\") pod \"test-model-fcd453-kserve-c78759c9-96pct\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") " pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:28.856408 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.856385    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-dshm\") pod \"test-model-fcd453-kserve-c78759c9-96pct\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") " pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:28.856688 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.856666    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4e29f82-02fc-42ba-a375-f8d9bc88123c-tls-certs\") pod \"test-model-fcd453-kserve-c78759c9-96pct\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") " pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:28.862185 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.862127    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5lxp\" (UniqueName: \"kubernetes.io/projected/c4e29f82-02fc-42ba-a375-f8d9bc88123c-kube-api-access-l5lxp\") pod \"test-model-fcd453-kserve-c78759c9-96pct\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") " pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:28.922396 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:28.922356    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:29.196035 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:29.196002    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"]
Sep 15 07:39:29.198423 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:39:29.198397    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4e29f82_02fc_42ba_a375_f8d9bc88123c.slice/crio-07f842df6d529d4af6fe6eae4573f6c8bb0f9fee65de75748737c52a56b1338a WatchSource:0}: Error finding container 07f842df6d529d4af6fe6eae4573f6c8bb0f9fee65de75748737c52a56b1338a: Status 404 returned error can't find the container with id 07f842df6d529d4af6fe6eae4573f6c8bb0f9fee65de75748737c52a56b1338a
Sep 15 07:39:29.717860 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:29.717817    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct" event={"ID":"c4e29f82-02fc-42ba-a375-f8d9bc88123c","Type":"ContainerStarted","Data":"18a89f96ae1df6a672467a5370b1359744a0934e400faf3a9d5b8f498c56e08f"}
Sep 15 07:39:29.718243 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:29.717869    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct" event={"ID":"c4e29f82-02fc-42ba-a375-f8d9bc88123c","Type":"ContainerStarted","Data":"07f842df6d529d4af6fe6eae4573f6c8bb0f9fee65de75748737c52a56b1338a"}
Sep 15 07:39:38.923078 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:38.923041    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:38.923078 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:38.923083    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:38.935772 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:38.935746    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:38.953824 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:38.953778    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct" podStartSLOduration=10.953764093 podStartE2EDuration="10.953764093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:39:29.740165795 +0000 UTC m=+1274.286735914" watchObservedRunningTime="2026-09-15 07:39:38.953764093 +0000 UTC m=+1283.500334190"
Sep 15 07:39:39.783453 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:39.783423    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:39:52.602575 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:52.602539    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp"]
Sep 15 07:39:52.605876 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:52.605857    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp"
Sep 15 07:39:52.608413 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:52.608390    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-526b4727\""
Sep 15 07:39:52.608541 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:52.608475    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-526b4727-dockercfg-cjv9g\""
Sep 15 07:39:52.622162 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:52.622126    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp"]
Sep 15 07:39:52.686361 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:52.686323    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18d7359a-3acd-4faa-94a4-baf155853f33-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp\" (UID: \"18d7359a-3acd-4faa-94a4-baf155853f33\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp"
Sep 15 07:39:52.686516 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:52.686396    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqf2r\" (UniqueName: \"kubernetes.io/projected/18d7359a-3acd-4faa-94a4-baf155853f33-kube-api-access-kqf2r\") pod \"payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp\" (UID: \"18d7359a-3acd-4faa-94a4-baf155853f33\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp"
Sep 15 07:39:52.787154 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:52.787109    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18d7359a-3acd-4faa-94a4-baf155853f33-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp\" (UID: \"18d7359a-3acd-4faa-94a4-baf155853f33\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp"
Sep 15 07:39:52.787364 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:52.787216    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqf2r\" (UniqueName: \"kubernetes.io/projected/18d7359a-3acd-4faa-94a4-baf155853f33-kube-api-access-kqf2r\") pod \"payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp\" (UID: \"18d7359a-3acd-4faa-94a4-baf155853f33\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp"
Sep 15 07:39:52.787887 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:52.787866    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18d7359a-3acd-4faa-94a4-baf155853f33-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp\" (UID: \"18d7359a-3acd-4faa-94a4-baf155853f33\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp"
Sep 15 07:39:52.795148 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:52.795123    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqf2r\" (UniqueName: \"kubernetes.io/projected/18d7359a-3acd-4faa-94a4-baf155853f33-kube-api-access-kqf2r\") pod \"payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp\" (UID: \"18d7359a-3acd-4faa-94a4-baf155853f33\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp"
Sep 15 07:39:52.918603 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:52.918508    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp"
Sep 15 07:39:53.189536 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:53.189502    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp"]
Sep 15 07:39:53.191419 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:39:53.191391    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d7359a_3acd_4faa_94a4_baf155853f33.slice/crio-2a00792407799d8c19918adf02c1711bc57884e53c9fd2fb22729709ff19760d WatchSource:0}: Error finding container 2a00792407799d8c19918adf02c1711bc57884e53c9fd2fb22729709ff19760d: Status 404 returned error can't find the container with id 2a00792407799d8c19918adf02c1711bc57884e53c9fd2fb22729709ff19760d
Sep 15 07:39:53.850971 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:53.850936    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp" event={"ID":"18d7359a-3acd-4faa-94a4-baf155853f33","Type":"ContainerStarted","Data":"ce5993764f6c0474365824892d2db45ec9948d2bc81fcea1671ba00f1a046699"}
Sep 15 07:39:53.851379 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:53.850994    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp"
Sep 15 07:39:53.851379 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:53.851012    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp" event={"ID":"18d7359a-3acd-4faa-94a4-baf155853f33","Type":"ContainerStarted","Data":"2a00792407799d8c19918adf02c1711bc57884e53c9fd2fb22729709ff19760d"}
Sep 15 07:39:53.869263 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:39:53.869214    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp" podStartSLOduration=1.8691983410000002 podStartE2EDuration="1.869198341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:39:53.86693131 +0000 UTC m=+1298.413501433" watchObservedRunningTime="2026-09-15 07:39:53.869198341 +0000 UTC m=+1298.415768440"
Sep 15 07:40:04.858774 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:04.858740    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp"
Sep 15 07:40:25.568205 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:25.568167    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-526b4727-t7vx8"]
Sep 15 07:40:25.571974 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:25.571957    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-526b4727-t7vx8"
Sep 15 07:40:25.574440 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:25.574418    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 07:40:25.574555 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:25.574440    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8jpwk\""
Sep 15 07:40:25.578820 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:25.578793    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-526b4727-t7vx8"]
Sep 15 07:40:25.606822 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:25.606789    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cee6ac6-2dc5-4140-8997-a0a06b92453f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-526b4727-t7vx8\" (UID: \"0cee6ac6-2dc5-4140-8997-a0a06b92453f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-526b4727-t7vx8"
Sep 15 07:40:25.707761 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:25.707728    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cee6ac6-2dc5-4140-8997-a0a06b92453f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-526b4727-t7vx8\" (UID: \"0cee6ac6-2dc5-4140-8997-a0a06b92453f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-526b4727-t7vx8"
Sep 15 07:40:25.708342 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:25.708323    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cee6ac6-2dc5-4140-8997-a0a06b92453f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-526b4727-t7vx8\" (UID: \"0cee6ac6-2dc5-4140-8997-a0a06b92453f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-526b4727-t7vx8"
Sep 15 07:40:25.884999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:25.884910    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-526b4727-t7vx8"
Sep 15 07:40:26.159967 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:26.159927    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-526b4727-t7vx8"]
Sep 15 07:40:26.162505 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:40:26.162455    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cee6ac6_2dc5_4140_8997_a0a06b92453f.slice/crio-87cf78861f13c134c16fd1da065165d3b1f89418e5b9267abb41391fc92bd88d WatchSource:0}: Error finding container 87cf78861f13c134c16fd1da065165d3b1f89418e5b9267abb41391fc92bd88d: Status 404 returned error can't find the container with id 87cf78861f13c134c16fd1da065165d3b1f89418e5b9267abb41391fc92bd88d
Sep 15 07:40:27.030390 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:27.030352    2571 generic.go:356] "Generic (PLEG): container finished" podID="0cee6ac6-2dc5-4140-8997-a0a06b92453f" containerID="faba866db02fefe5f003c108590b572aed495491b9bb656a4068defe05cb426c" exitCode=0
Sep 15 07:40:27.030772 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:27.030448    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-526b4727-t7vx8" event={"ID":"0cee6ac6-2dc5-4140-8997-a0a06b92453f","Type":"ContainerDied","Data":"faba866db02fefe5f003c108590b572aed495491b9bb656a4068defe05cb426c"}
Sep 15 07:40:27.030772 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:27.030508    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-526b4727-t7vx8" event={"ID":"0cee6ac6-2dc5-4140-8997-a0a06b92453f","Type":"ContainerStarted","Data":"87cf78861f13c134c16fd1da065165d3b1f89418e5b9267abb41391fc92bd88d"}
Sep 15 07:40:28.172974 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:28.172951    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-526b4727-t7vx8"
Sep 15 07:40:28.233871 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:28.233826    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cee6ac6-2dc5-4140-8997-a0a06b92453f-maas-api-service-ca\") pod \"0cee6ac6-2dc5-4140-8997-a0a06b92453f\" (UID: \"0cee6ac6-2dc5-4140-8997-a0a06b92453f\") "
Sep 15 07:40:28.234145 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:28.234122    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cee6ac6-2dc5-4140-8997-a0a06b92453f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0cee6ac6-2dc5-4140-8997-a0a06b92453f" (UID: "0cee6ac6-2dc5-4140-8997-a0a06b92453f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:40:28.335035 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:28.334953    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cee6ac6-2dc5-4140-8997-a0a06b92453f-maas-api-service-ca\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:40:29.042922 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:29.042888    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-526b4727-t7vx8" event={"ID":"0cee6ac6-2dc5-4140-8997-a0a06b92453f","Type":"ContainerDied","Data":"87cf78861f13c134c16fd1da065165d3b1f89418e5b9267abb41391fc92bd88d"}
Sep 15 07:40:29.042922 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:29.042922    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-526b4727-t7vx8"
Sep 15 07:40:29.043199 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:29.042937    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87cf78861f13c134c16fd1da065165d3b1f89418e5b9267abb41391fc92bd88d"
Sep 15 07:40:36.142456 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:36.142423    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp"]
Sep 15 07:40:36.143528 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:36.142633    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp" podUID="18d7359a-3acd-4faa-94a4-baf155853f33" containerName="payload-pre-processing" containerID="cri-o://ce5993764f6c0474365824892d2db45ec9948d2bc81fcea1671ba00f1a046699" gracePeriod=30
Sep 15 07:40:36.305479 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:36.305444    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-526b4727-t7vx8"]
Sep 15 07:40:36.307412 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:36.307387    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-526b4727-t7vx8"]
Sep 15 07:40:36.401103 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:36.401038    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp"
Sep 15 07:40:36.517550 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:36.517514    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18d7359a-3acd-4faa-94a4-baf155853f33-plugins-config-volume\") pod \"18d7359a-3acd-4faa-94a4-baf155853f33\" (UID: \"18d7359a-3acd-4faa-94a4-baf155853f33\") "
Sep 15 07:40:36.517713 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:36.517615    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqf2r\" (UniqueName: \"kubernetes.io/projected/18d7359a-3acd-4faa-94a4-baf155853f33-kube-api-access-kqf2r\") pod \"18d7359a-3acd-4faa-94a4-baf155853f33\" (UID: \"18d7359a-3acd-4faa-94a4-baf155853f33\") "
Sep 15 07:40:36.517895 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:36.517871    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18d7359a-3acd-4faa-94a4-baf155853f33-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "18d7359a-3acd-4faa-94a4-baf155853f33" (UID: "18d7359a-3acd-4faa-94a4-baf155853f33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:40:36.519595 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:36.519575    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d7359a-3acd-4faa-94a4-baf155853f33-kube-api-access-kqf2r" (OuterVolumeSpecName: "kube-api-access-kqf2r") pod "18d7359a-3acd-4faa-94a4-baf155853f33" (UID: "18d7359a-3acd-4faa-94a4-baf155853f33"). InnerVolumeSpecName "kube-api-access-kqf2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:40:36.618785 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:36.618748    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18d7359a-3acd-4faa-94a4-baf155853f33-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:40:36.618785 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:36.618780    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqf2r\" (UniqueName: \"kubernetes.io/projected/18d7359a-3acd-4faa-94a4-baf155853f33-kube-api-access-kqf2r\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:40:37.090592 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:37.090558    2571 generic.go:356] "Generic (PLEG): container finished" podID="18d7359a-3acd-4faa-94a4-baf155853f33" containerID="ce5993764f6c0474365824892d2db45ec9948d2bc81fcea1671ba00f1a046699" exitCode=0
Sep 15 07:40:37.090592 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:37.090598    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp" event={"ID":"18d7359a-3acd-4faa-94a4-baf155853f33","Type":"ContainerDied","Data":"ce5993764f6c0474365824892d2db45ec9948d2bc81fcea1671ba00f1a046699"}
Sep 15 07:40:37.090805 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:37.090604    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp"
Sep 15 07:40:37.090805 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:37.090637    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp" event={"ID":"18d7359a-3acd-4faa-94a4-baf155853f33","Type":"ContainerDied","Data":"2a00792407799d8c19918adf02c1711bc57884e53c9fd2fb22729709ff19760d"}
Sep 15 07:40:37.090805 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:37.090659    2571 scope.go:117] "RemoveContainer" containerID="ce5993764f6c0474365824892d2db45ec9948d2bc81fcea1671ba00f1a046699"
Sep 15 07:40:37.100341 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:37.100322    2571 scope.go:117] "RemoveContainer" containerID="ce5993764f6c0474365824892d2db45ec9948d2bc81fcea1671ba00f1a046699"
Sep 15 07:40:37.100626 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:40:37.100608    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce5993764f6c0474365824892d2db45ec9948d2bc81fcea1671ba00f1a046699\": container with ID starting with ce5993764f6c0474365824892d2db45ec9948d2bc81fcea1671ba00f1a046699 not found: ID does not exist" containerID="ce5993764f6c0474365824892d2db45ec9948d2bc81fcea1671ba00f1a046699"
Sep 15 07:40:37.100674 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:37.100635    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce5993764f6c0474365824892d2db45ec9948d2bc81fcea1671ba00f1a046699"} err="failed to get container status \"ce5993764f6c0474365824892d2db45ec9948d2bc81fcea1671ba00f1a046699\": rpc error: code = NotFound desc = could not find container \"ce5993764f6c0474365824892d2db45ec9948d2bc81fcea1671ba00f1a046699\": container with ID starting with ce5993764f6c0474365824892d2db45ec9948d2bc81fcea1671ba00f1a046699 not found: ID does not exist"
Sep 15 07:40:37.118641 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:37.118603    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp"]
Sep 15 07:40:37.123149 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:37.123123    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-526b4727-766f88884b-d6cwp"]
Sep 15 07:40:37.909556 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:37.909522    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cee6ac6-2dc5-4140-8997-a0a06b92453f" path="/var/lib/kubelet/pods/0cee6ac6-2dc5-4140-8997-a0a06b92453f/volumes"
Sep 15 07:40:37.909933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:40:37.909916    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18d7359a-3acd-4faa-94a4-baf155853f33" path="/var/lib/kubelet/pods/18d7359a-3acd-4faa-94a4-baf155853f33/volumes"
Sep 15 07:41:37.760016 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.759980    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd"]
Sep 15 07:41:37.760481 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.760467    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18d7359a-3acd-4faa-94a4-baf155853f33" containerName="payload-pre-processing"
Sep 15 07:41:37.760481 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.760481    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="18d7359a-3acd-4faa-94a4-baf155853f33" containerName="payload-pre-processing"
Sep 15 07:41:37.760559 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.760505    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cee6ac6-2dc5-4140-8997-a0a06b92453f" containerName="revoke-keys"
Sep 15 07:41:37.760559 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.760511    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cee6ac6-2dc5-4140-8997-a0a06b92453f" containerName="revoke-keys"
Sep 15 07:41:37.760625 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.760570    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cee6ac6-2dc5-4140-8997-a0a06b92453f" containerName="revoke-keys"
Sep 15 07:41:37.760625 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.760582    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="18d7359a-3acd-4faa-94a4-baf155853f33" containerName="payload-pre-processing"
Sep 15 07:41:37.764890 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.764871    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd"
Sep 15 07:41:37.767607 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.767580    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-27b9d8\""
Sep 15 07:41:37.774127 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.774101    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd"]
Sep 15 07:41:37.789264 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.789220    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2"]
Sep 15 07:41:37.793721 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.793697    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2"
Sep 15 07:41:37.796764 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.796741    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-27b9d8-dockercfg-8jtpr\""
Sep 15 07:41:37.802514 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.802493    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2"]
Sep 15 07:41:37.830509 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.830471    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f1cb03-c1e7-4820-934d-62efeb27d9ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd\" (UID: \"60f1cb03-c1e7-4820-934d-62efeb27d9ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd"
Sep 15 07:41:37.830656 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.830520    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c672cec7-2811-4072-961f-9dd921e5aef5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2\" (UID: \"c672cec7-2811-4072-961f-9dd921e5aef5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2"
Sep 15 07:41:37.830698 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.830661    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk2kb\" (UniqueName: \"kubernetes.io/projected/60f1cb03-c1e7-4820-934d-62efeb27d9ea-kube-api-access-gk2kb\") pod \"payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd\" (UID: \"60f1cb03-c1e7-4820-934d-62efeb27d9ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd"
Sep 15 07:41:37.830738 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.830701    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l592s\" (UniqueName: \"kubernetes.io/projected/c672cec7-2811-4072-961f-9dd921e5aef5-kube-api-access-l592s\") pod \"payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2\" (UID: \"c672cec7-2811-4072-961f-9dd921e5aef5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2"
Sep 15 07:41:37.932309 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.932246    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f1cb03-c1e7-4820-934d-62efeb27d9ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd\" (UID: \"60f1cb03-c1e7-4820-934d-62efeb27d9ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd"
Sep 15 07:41:37.932309 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.932313    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c672cec7-2811-4072-961f-9dd921e5aef5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2\" (UID: \"c672cec7-2811-4072-961f-9dd921e5aef5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2"
Sep 15 07:41:37.932514 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.932391    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk2kb\" (UniqueName: \"kubernetes.io/projected/60f1cb03-c1e7-4820-934d-62efeb27d9ea-kube-api-access-gk2kb\") pod \"payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd\" (UID: \"60f1cb03-c1e7-4820-934d-62efeb27d9ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd"
Sep 15 07:41:37.932514 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.932419    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l592s\" (UniqueName: \"kubernetes.io/projected/c672cec7-2811-4072-961f-9dd921e5aef5-kube-api-access-l592s\") pod \"payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2\" (UID: \"c672cec7-2811-4072-961f-9dd921e5aef5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2"
Sep 15 07:41:37.932969 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.932942    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f1cb03-c1e7-4820-934d-62efeb27d9ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd\" (UID: \"60f1cb03-c1e7-4820-934d-62efeb27d9ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd"
Sep 15 07:41:37.933050 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.933028    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c672cec7-2811-4072-961f-9dd921e5aef5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2\" (UID: \"c672cec7-2811-4072-961f-9dd921e5aef5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2"
Sep 15 07:41:37.940031 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.940002    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk2kb\" (UniqueName: \"kubernetes.io/projected/60f1cb03-c1e7-4820-934d-62efeb27d9ea-kube-api-access-gk2kb\") pod \"payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd\" (UID: \"60f1cb03-c1e7-4820-934d-62efeb27d9ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd"
Sep 15 07:41:37.940430 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:37.940413    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l592s\" (UniqueName: \"kubernetes.io/projected/c672cec7-2811-4072-961f-9dd921e5aef5-kube-api-access-l592s\") pod \"payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2\" (UID: \"c672cec7-2811-4072-961f-9dd921e5aef5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2"
Sep 15 07:41:38.076359 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:38.076272    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd"
Sep 15 07:41:38.105649 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:38.105618    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2"
Sep 15 07:41:38.349460 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:38.349231    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd"]
Sep 15 07:41:38.352211 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:41:38.352184    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f1cb03_c1e7_4820_934d_62efeb27d9ea.slice/crio-ddb9fddfb342dd878b26fecfe2c947010b70c913d573e60896d52d78820d736c WatchSource:0}: Error finding container ddb9fddfb342dd878b26fecfe2c947010b70c913d573e60896d52d78820d736c: Status 404 returned error can't find the container with id ddb9fddfb342dd878b26fecfe2c947010b70c913d573e60896d52d78820d736c
Sep 15 07:41:38.385065 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:38.385038    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2"]
Sep 15 07:41:38.386626 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:41:38.386600    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc672cec7_2811_4072_961f_9dd921e5aef5.slice/crio-b9a78b339a353a99290c70570c9c6ca3e803b3996a8160e7d538e2f02aea9a58 WatchSource:0}: Error finding container b9a78b339a353a99290c70570c9c6ca3e803b3996a8160e7d538e2f02aea9a58: Status 404 returned error can't find the container with id b9a78b339a353a99290c70570c9c6ca3e803b3996a8160e7d538e2f02aea9a58
Sep 15 07:41:38.429633 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:38.429607    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd" event={"ID":"60f1cb03-c1e7-4820-934d-62efeb27d9ea","Type":"ContainerStarted","Data":"284a06d0d4fdb049655702c104c7846605edbec03291d484be6ea8011bf63a0d"}
Sep 15 07:41:38.429737 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:38.429658    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd" event={"ID":"60f1cb03-c1e7-4820-934d-62efeb27d9ea","Type":"ContainerStarted","Data":"ddb9fddfb342dd878b26fecfe2c947010b70c913d573e60896d52d78820d736c"}
Sep 15 07:41:38.431134 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:38.431110    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2" event={"ID":"c672cec7-2811-4072-961f-9dd921e5aef5","Type":"ContainerStarted","Data":"b9a78b339a353a99290c70570c9c6ca3e803b3996a8160e7d538e2f02aea9a58"}
Sep 15 07:41:39.439587 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:39.439551    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2" event={"ID":"c672cec7-2811-4072-961f-9dd921e5aef5","Type":"ContainerStarted","Data":"26c5bcbc02c4eeb1989a9e4621b3824e8d6faea186665d5348b0bd7789865700"}
Sep 15 07:41:39.440027 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:39.439696    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2"
Sep 15 07:41:39.457691 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:39.457638    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2" podStartSLOduration=2.457619705 podStartE2EDuration="2.457619705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:41:39.457338799 +0000 UTC m=+1404.003908924" watchObservedRunningTime="2026-09-15 07:41:39.457619705 +0000 UTC m=+1404.004189805"
Sep 15 07:41:39.474948 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:39.474899    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd" podStartSLOduration=2.4748845680000002 podStartE2EDuration="2.474884568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:41:39.472398091 +0000 UTC m=+1404.018968217" watchObservedRunningTime="2026-09-15 07:41:39.474884568 +0000 UTC m=+1404.021454667"
Sep 15 07:41:40.445774 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:40.445732    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd"
Sep 15 07:41:50.448348 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:50.448313    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2"
Sep 15 07:41:51.452882 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:41:51.452849    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd"
Sep 15 07:42:16.000423 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.000339    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"]
Sep 15 07:42:16.004009 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.003983    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:16.006533 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.006516    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f56165\"/\"e2e-conflict-a-30d020fe-kserve-self-signed-certs\""
Sep 15 07:42:16.013557 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.013535    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"]
Sep 15 07:42:16.121354 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.121316    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-tmp-dir\") pod \"e2e-conflict-a-30d020fe-kserve-64855995df-ssklk\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:16.121535 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.121384    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-dshm\") pod \"e2e-conflict-a-30d020fe-kserve-64855995df-ssklk\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:16.121535 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.121444    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-home\") pod \"e2e-conflict-a-30d020fe-kserve-64855995df-ssklk\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:16.121535 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.121498    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-model-cache\") pod \"e2e-conflict-a-30d020fe-kserve-64855995df-ssklk\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:16.121676 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.121537    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqmk2\" (UniqueName: \"kubernetes.io/projected/16abe562-4784-49d5-9b9a-c46ffc4900d2-kube-api-access-kqmk2\") pod \"e2e-conflict-a-30d020fe-kserve-64855995df-ssklk\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:16.121676 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.121600    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16abe562-4784-49d5-9b9a-c46ffc4900d2-tls-certs\") pod \"e2e-conflict-a-30d020fe-kserve-64855995df-ssklk\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:16.222941 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.222905    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16abe562-4784-49d5-9b9a-c46ffc4900d2-tls-certs\") pod \"e2e-conflict-a-30d020fe-kserve-64855995df-ssklk\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:16.223118 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.222956    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-tmp-dir\") pod \"e2e-conflict-a-30d020fe-kserve-64855995df-ssklk\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:16.223118 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.222997    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-dshm\") pod \"e2e-conflict-a-30d020fe-kserve-64855995df-ssklk\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:16.223118 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.223028    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-home\") pod \"e2e-conflict-a-30d020fe-kserve-64855995df-ssklk\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:16.223118 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.223048    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-model-cache\") pod \"e2e-conflict-a-30d020fe-kserve-64855995df-ssklk\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:16.223349 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.223275    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqmk2\" (UniqueName: \"kubernetes.io/projected/16abe562-4784-49d5-9b9a-c46ffc4900d2-kube-api-access-kqmk2\") pod \"e2e-conflict-a-30d020fe-kserve-64855995df-ssklk\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:16.223496 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.223465    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-home\") pod \"e2e-conflict-a-30d020fe-kserve-64855995df-ssklk\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:16.223627 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.223497    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-model-cache\") pod \"e2e-conflict-a-30d020fe-kserve-64855995df-ssklk\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:16.223627 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.223531    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-tmp-dir\") pod \"e2e-conflict-a-30d020fe-kserve-64855995df-ssklk\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:16.225429 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.225407    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-dshm\") pod \"e2e-conflict-a-30d020fe-kserve-64855995df-ssklk\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:16.225525 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.225442    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16abe562-4784-49d5-9b9a-c46ffc4900d2-tls-certs\") pod \"e2e-conflict-a-30d020fe-kserve-64855995df-ssklk\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:16.231232 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.231207    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqmk2\" (UniqueName: \"kubernetes.io/projected/16abe562-4784-49d5-9b9a-c46ffc4900d2-kube-api-access-kqmk2\") pod \"e2e-conflict-a-30d020fe-kserve-64855995df-ssklk\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:16.314334 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.314231    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:16.588467 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.588440    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"]
Sep 15 07:42:16.590772 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:42:16.590745    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16abe562_4784_49d5_9b9a_c46ffc4900d2.slice/crio-da6c9243aee8e2f67d68482a6582b0f6b1ec7c2102a207e73046bccc74e1a6f9 WatchSource:0}: Error finding container da6c9243aee8e2f67d68482a6582b0f6b1ec7c2102a207e73046bccc74e1a6f9: Status 404 returned error can't find the container with id da6c9243aee8e2f67d68482a6582b0f6b1ec7c2102a207e73046bccc74e1a6f9
Sep 15 07:42:16.648566 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:16.648533    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk" event={"ID":"16abe562-4784-49d5-9b9a-c46ffc4900d2","Type":"ContainerStarted","Data":"da6c9243aee8e2f67d68482a6582b0f6b1ec7c2102a207e73046bccc74e1a6f9"}
Sep 15 07:42:17.655344 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:17.655304    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk" event={"ID":"16abe562-4784-49d5-9b9a-c46ffc4900d2","Type":"ContainerStarted","Data":"b7e65dae342474c7ffbd34c0a0b6ee67b445a838efed2a81d449353de90d116b"}
Sep 15 07:42:17.672977 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:17.672924    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk" podStartSLOduration=2.672910612 podStartE2EDuration="2.672910612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:42:17.671743794 +0000 UTC m=+1442.218313919" watchObservedRunningTime="2026-09-15 07:42:17.672910612 +0000 UTC m=+1442.219480711"
Sep 15 07:42:26.315382 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:26.315329    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:26.315879 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:26.315414    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:26.328302 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:26.328273    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:26.716560 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:26.716530    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:42:35.794248 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:35.794208    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"]
Sep 15 07:42:38.303906 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:38.303871    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"]
Sep 15 07:42:38.437566 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:38.437531    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee0096-5kmd4"]
Sep 15 07:42:38.441654 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:38.441638    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee0096-5kmd4"
Sep 15 07:42:38.443724 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:38.443704    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8jpwk\""
Sep 15 07:42:38.443990 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:38.443972    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 07:42:38.447943 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:38.447924    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee0096-5kmd4"]
Sep 15 07:42:38.548316 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:38.548247    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83d162c4-ae04-4957-8775-41dbe5ce03c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ee0096-5kmd4\" (UID: \"83d162c4-ae04-4957-8775-41dbe5ce03c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee0096-5kmd4"
Sep 15 07:42:38.649156 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:38.649069    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83d162c4-ae04-4957-8775-41dbe5ce03c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ee0096-5kmd4\" (UID: \"83d162c4-ae04-4957-8775-41dbe5ce03c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee0096-5kmd4"
Sep 15 07:42:38.649791 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:38.649769    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83d162c4-ae04-4957-8775-41dbe5ce03c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ee0096-5kmd4\" (UID: \"83d162c4-ae04-4957-8775-41dbe5ce03c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee0096-5kmd4"
Sep 15 07:42:38.755501 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:38.755464    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee0096-5kmd4"
Sep 15 07:42:39.029877 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:39.029842    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee0096-5kmd4"]
Sep 15 07:42:39.032534 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:42:39.032508    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d162c4_ae04_4957_8775_41dbe5ce03c5.slice/crio-64fd635a93ecec4237d2ef40ba6ef46426e6d183bb9a65791be273dcc08c1f62 WatchSource:0}: Error finding container 64fd635a93ecec4237d2ef40ba6ef46426e6d183bb9a65791be273dcc08c1f62: Status 404 returned error can't find the container with id 64fd635a93ecec4237d2ef40ba6ef46426e6d183bb9a65791be273dcc08c1f62
Sep 15 07:42:39.793592 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:39.793555    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee0096-5kmd4" event={"ID":"83d162c4-ae04-4957-8775-41dbe5ce03c5","Type":"ContainerStarted","Data":"ae429ffe0e5fd9fae0c1f590f82ea356751e8b1ec94287c4a805d48f52f3085b"}
Sep 15 07:42:39.794063 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:39.793611    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee0096-5kmd4" event={"ID":"83d162c4-ae04-4957-8775-41dbe5ce03c5","Type":"ContainerStarted","Data":"64fd635a93ecec4237d2ef40ba6ef46426e6d183bb9a65791be273dcc08c1f62"}
Sep 15 07:42:39.809028 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:39.808981    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee0096-5kmd4" podStartSLOduration=1.808967148 podStartE2EDuration="1.808967148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:42:39.807143769 +0000 UTC m=+1464.353713885" watchObservedRunningTime="2026-09-15 07:42:39.808967148 +0000 UTC m=+1464.355537246"
Sep 15 07:42:40.801075 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:40.801038    2571 generic.go:356] "Generic (PLEG): container finished" podID="83d162c4-ae04-4957-8775-41dbe5ce03c5" containerID="ae429ffe0e5fd9fae0c1f590f82ea356751e8b1ec94287c4a805d48f52f3085b" exitCode=0
Sep 15 07:42:40.801494 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:40.801121    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee0096-5kmd4" event={"ID":"83d162c4-ae04-4957-8775-41dbe5ce03c5","Type":"ContainerDied","Data":"ae429ffe0e5fd9fae0c1f590f82ea356751e8b1ec94287c4a805d48f52f3085b"}
Sep 15 07:42:41.950894 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:41.950870    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee0096-5kmd4"
Sep 15 07:42:41.979481 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:41.979443    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83d162c4-ae04-4957-8775-41dbe5ce03c5-maas-api-service-ca\") pod \"83d162c4-ae04-4957-8775-41dbe5ce03c5\" (UID: \"83d162c4-ae04-4957-8775-41dbe5ce03c5\") "
Sep 15 07:42:41.979764 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:41.979742    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83d162c4-ae04-4957-8775-41dbe5ce03c5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83d162c4-ae04-4957-8775-41dbe5ce03c5" (UID: "83d162c4-ae04-4957-8775-41dbe5ce03c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:42:41.979900 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:41.979883    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83d162c4-ae04-4957-8775-41dbe5ce03c5-maas-api-service-ca\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:42:42.813269 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:42.813233    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee0096-5kmd4" event={"ID":"83d162c4-ae04-4957-8775-41dbe5ce03c5","Type":"ContainerDied","Data":"64fd635a93ecec4237d2ef40ba6ef46426e6d183bb9a65791be273dcc08c1f62"}
Sep 15 07:42:42.813441 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:42.813286    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee0096-5kmd4"
Sep 15 07:42:42.813441 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:42.813293    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64fd635a93ecec4237d2ef40ba6ef46426e6d183bb9a65791be273dcc08c1f62"
Sep 15 07:42:50.813736 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:50.813675    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm" podUID="d3f34c4a-b956-4c19-aaee-f85fdef80b99" containerName="main" containerID="cri-o://15a62fa8ff3e8e8f9eac5b59e1ec6563d9b9e3bebf8d279ad8aa8e336e50fb00" gracePeriod=45
Sep 15 07:42:51.076534 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.076464    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:42:51.168776 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.168745    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-home\") pod \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") "
Sep 15 07:42:51.168950 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.168808    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f34c4a-b956-4c19-aaee-f85fdef80b99-tls-certs\") pod \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") "
Sep 15 07:42:51.168950 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.168879    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h95rf\" (UniqueName: \"kubernetes.io/projected/d3f34c4a-b956-4c19-aaee-f85fdef80b99-kube-api-access-h95rf\") pod \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") "
Sep 15 07:42:51.168950 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.168938    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-dshm\") pod \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") "
Sep 15 07:42:51.169137 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.168972    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-model-cache\") pod \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") "
Sep 15 07:42:51.169137 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.169024    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-tmp-dir\") pod \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\" (UID: \"d3f34c4a-b956-4c19-aaee-f85fdef80b99\") "
Sep 15 07:42:51.169137 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.169017    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-home" (OuterVolumeSpecName: "home") pod "d3f34c4a-b956-4c19-aaee-f85fdef80b99" (UID: "d3f34c4a-b956-4c19-aaee-f85fdef80b99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:42:51.169328 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.169296    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-model-cache" (OuterVolumeSpecName: "model-cache") pod "d3f34c4a-b956-4c19-aaee-f85fdef80b99" (UID: "d3f34c4a-b956-4c19-aaee-f85fdef80b99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:42:51.169328 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.169315    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d3f34c4a-b956-4c19-aaee-f85fdef80b99" (UID: "d3f34c4a-b956-4c19-aaee-f85fdef80b99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:42:51.169463 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.169440    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-model-cache\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:42:51.169463 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.169465    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-tmp-dir\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:42:51.169619 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.169478    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-home\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:42:51.171113 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.171091    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-dshm" (OuterVolumeSpecName: "dshm") pod "d3f34c4a-b956-4c19-aaee-f85fdef80b99" (UID: "d3f34c4a-b956-4c19-aaee-f85fdef80b99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:42:51.171193 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.171162    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f34c4a-b956-4c19-aaee-f85fdef80b99-kube-api-access-h95rf" (OuterVolumeSpecName: "kube-api-access-h95rf") pod "d3f34c4a-b956-4c19-aaee-f85fdef80b99" (UID: "d3f34c4a-b956-4c19-aaee-f85fdef80b99"). InnerVolumeSpecName "kube-api-access-h95rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:42:51.171238 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.171207    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3f34c4a-b956-4c19-aaee-f85fdef80b99-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3f34c4a-b956-4c19-aaee-f85fdef80b99" (UID: "d3f34c4a-b956-4c19-aaee-f85fdef80b99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:42:51.270468 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.270416    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f34c4a-b956-4c19-aaee-f85fdef80b99-tls-certs\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:42:51.270468 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.270462    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h95rf\" (UniqueName: \"kubernetes.io/projected/d3f34c4a-b956-4c19-aaee-f85fdef80b99-kube-api-access-h95rf\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:42:51.270468 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.270475    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f34c4a-b956-4c19-aaee-f85fdef80b99-dshm\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:42:51.713294 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.713237    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg"]
Sep 15 07:42:51.713596 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.713539    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg" podUID="c6d1224c-ff36-4220-ab57-63666a719501" containerName="payload-pre-processing" containerID="cri-o://a95c227bb1f6dad9b36947ea34d02a5ccfa26206b6b55ac2fbf21fc0fe85c760" gracePeriod=30
Sep 15 07:42:51.868618 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.868583    2571 generic.go:356] "Generic (PLEG): container finished" podID="c6d1224c-ff36-4220-ab57-63666a719501" containerID="a95c227bb1f6dad9b36947ea34d02a5ccfa26206b6b55ac2fbf21fc0fe85c760" exitCode=0
Sep 15 07:42:51.869054 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.868660    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg" event={"ID":"c6d1224c-ff36-4220-ab57-63666a719501","Type":"ContainerDied","Data":"a95c227bb1f6dad9b36947ea34d02a5ccfa26206b6b55ac2fbf21fc0fe85c760"}
Sep 15 07:42:51.870036 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.870009    2571 generic.go:356] "Generic (PLEG): container finished" podID="d3f34c4a-b956-4c19-aaee-f85fdef80b99" containerID="15a62fa8ff3e8e8f9eac5b59e1ec6563d9b9e3bebf8d279ad8aa8e336e50fb00" exitCode=0
Sep 15 07:42:51.870163 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.870054    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"
Sep 15 07:42:51.870163 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.870092    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm" event={"ID":"d3f34c4a-b956-4c19-aaee-f85fdef80b99","Type":"ContainerDied","Data":"15a62fa8ff3e8e8f9eac5b59e1ec6563d9b9e3bebf8d279ad8aa8e336e50fb00"}
Sep 15 07:42:51.870163 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.870136    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm" event={"ID":"d3f34c4a-b956-4c19-aaee-f85fdef80b99","Type":"ContainerDied","Data":"4b7cb2708bea3fa875228ff9f563827af582a2b74f4c048e34c91babf7a85a93"}
Sep 15 07:42:51.870163 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.870157    2571 scope.go:117] "RemoveContainer" containerID="15a62fa8ff3e8e8f9eac5b59e1ec6563d9b9e3bebf8d279ad8aa8e336e50fb00"
Sep 15 07:42:51.880403 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.880185    2571 scope.go:117] "RemoveContainer" containerID="15a62fa8ff3e8e8f9eac5b59e1ec6563d9b9e3bebf8d279ad8aa8e336e50fb00"
Sep 15 07:42:51.880606 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:42:51.880579    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15a62fa8ff3e8e8f9eac5b59e1ec6563d9b9e3bebf8d279ad8aa8e336e50fb00\": container with ID starting with 15a62fa8ff3e8e8f9eac5b59e1ec6563d9b9e3bebf8d279ad8aa8e336e50fb00 not found: ID does not exist" containerID="15a62fa8ff3e8e8f9eac5b59e1ec6563d9b9e3bebf8d279ad8aa8e336e50fb00"
Sep 15 07:42:51.880678 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.880619    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a62fa8ff3e8e8f9eac5b59e1ec6563d9b9e3bebf8d279ad8aa8e336e50fb00"} err="failed to get container status \"15a62fa8ff3e8e8f9eac5b59e1ec6563d9b9e3bebf8d279ad8aa8e336e50fb00\": rpc error: code = NotFound desc = could not find container \"15a62fa8ff3e8e8f9eac5b59e1ec6563d9b9e3bebf8d279ad8aa8e336e50fb00\": container with ID starting with 15a62fa8ff3e8e8f9eac5b59e1ec6563d9b9e3bebf8d279ad8aa8e336e50fb00 not found: ID does not exist"
Sep 15 07:42:51.893290 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.893264    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"]
Sep 15 07:42:51.900082 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.900056    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ee0096/test-model-ee0096-kserve-77f5bb9fd7-c5jpm"]
Sep 15 07:42:51.910963 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.910934    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3f34c4a-b956-4c19-aaee-f85fdef80b99" path="/var/lib/kubelet/pods/d3f34c4a-b956-4c19-aaee-f85fdef80b99/volumes"
Sep 15 07:42:51.931952 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.931925    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee0096-5kmd4"]
Sep 15 07:42:51.935803 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.935779    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee0096-5kmd4"]
Sep 15 07:42:51.965798 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:51.965747    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg"
Sep 15 07:42:52.078931 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:52.078893    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glmzq\" (UniqueName: \"kubernetes.io/projected/c6d1224c-ff36-4220-ab57-63666a719501-kube-api-access-glmzq\") pod \"c6d1224c-ff36-4220-ab57-63666a719501\" (UID: \"c6d1224c-ff36-4220-ab57-63666a719501\") "
Sep 15 07:42:52.079110 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:52.079013    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d1224c-ff36-4220-ab57-63666a719501-plugins-config-volume\") pod \"c6d1224c-ff36-4220-ab57-63666a719501\" (UID: \"c6d1224c-ff36-4220-ab57-63666a719501\") "
Sep 15 07:42:52.079417 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:52.079391    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d1224c-ff36-4220-ab57-63666a719501-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c6d1224c-ff36-4220-ab57-63666a719501" (UID: "c6d1224c-ff36-4220-ab57-63666a719501"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:42:52.080961 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:52.080941    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d1224c-ff36-4220-ab57-63666a719501-kube-api-access-glmzq" (OuterVolumeSpecName: "kube-api-access-glmzq") pod "c6d1224c-ff36-4220-ab57-63666a719501" (UID: "c6d1224c-ff36-4220-ab57-63666a719501"). InnerVolumeSpecName "kube-api-access-glmzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:42:52.180427 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:52.180390    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d1224c-ff36-4220-ab57-63666a719501-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:42:52.180427 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:52.180419    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glmzq\" (UniqueName: \"kubernetes.io/projected/c6d1224c-ff36-4220-ab57-63666a719501-kube-api-access-glmzq\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:42:52.877329 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:52.877286    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg" event={"ID":"c6d1224c-ff36-4220-ab57-63666a719501","Type":"ContainerDied","Data":"f9f4518237e0ff814370cc7cf16db4337e5caaf653d3e38632b5120c0dceaf38"}
Sep 15 07:42:52.877329 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:52.877313    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg"
Sep 15 07:42:52.877881 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:52.877364    2571 scope.go:117] "RemoveContainer" containerID="a95c227bb1f6dad9b36947ea34d02a5ccfa26206b6b55ac2fbf21fc0fe85c760"
Sep 15 07:42:52.899047 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:52.899019    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg"]
Sep 15 07:42:52.901103 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:52.901077    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ee0096-79986cd4c8-vhprg"]
Sep 15 07:42:53.323177 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.323136    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct" podUID="c4e29f82-02fc-42ba-a375-f8d9bc88123c" containerName="main" containerID="cri-o://18a89f96ae1df6a672467a5370b1359744a0934e400faf3a9d5b8f498c56e08f" gracePeriod=45
Sep 15 07:42:53.574975 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.574908    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:42:53.696802 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.696767    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-model-cache\") pod \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") "
Sep 15 07:42:53.696802 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.696805    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4e29f82-02fc-42ba-a375-f8d9bc88123c-tls-certs\") pod \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") "
Sep 15 07:42:53.697054 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.696852    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-tmp-dir\") pod \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") "
Sep 15 07:42:53.697054 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.696892    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5lxp\" (UniqueName: \"kubernetes.io/projected/c4e29f82-02fc-42ba-a375-f8d9bc88123c-kube-api-access-l5lxp\") pod \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") "
Sep 15 07:42:53.697054 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.696924    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-home\") pod \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") "
Sep 15 07:42:53.697054 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.697002    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-dshm\") pod \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\" (UID: \"c4e29f82-02fc-42ba-a375-f8d9bc88123c\") "
Sep 15 07:42:53.697246 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.697088    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-model-cache" (OuterVolumeSpecName: "model-cache") pod "c4e29f82-02fc-42ba-a375-f8d9bc88123c" (UID: "c4e29f82-02fc-42ba-a375-f8d9bc88123c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:42:53.697246 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.697176    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c4e29f82-02fc-42ba-a375-f8d9bc88123c" (UID: "c4e29f82-02fc-42ba-a375-f8d9bc88123c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:42:53.697362 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.697275    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-home" (OuterVolumeSpecName: "home") pod "c4e29f82-02fc-42ba-a375-f8d9bc88123c" (UID: "c4e29f82-02fc-42ba-a375-f8d9bc88123c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:42:53.697400 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.697386    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-model-cache\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:42:53.697400 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.697399    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-tmp-dir\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:42:53.697518 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.697408    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-home\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:42:53.699196 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.699167    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4e29f82-02fc-42ba-a375-f8d9bc88123c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c4e29f82-02fc-42ba-a375-f8d9bc88123c" (UID: "c4e29f82-02fc-42ba-a375-f8d9bc88123c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:42:53.699196 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.699189    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-dshm" (OuterVolumeSpecName: "dshm") pod "c4e29f82-02fc-42ba-a375-f8d9bc88123c" (UID: "c4e29f82-02fc-42ba-a375-f8d9bc88123c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:42:53.699349 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.699230    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e29f82-02fc-42ba-a375-f8d9bc88123c-kube-api-access-l5lxp" (OuterVolumeSpecName: "kube-api-access-l5lxp") pod "c4e29f82-02fc-42ba-a375-f8d9bc88123c" (UID: "c4e29f82-02fc-42ba-a375-f8d9bc88123c"). InnerVolumeSpecName "kube-api-access-l5lxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:42:53.798479 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.798447    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5lxp\" (UniqueName: \"kubernetes.io/projected/c4e29f82-02fc-42ba-a375-f8d9bc88123c-kube-api-access-l5lxp\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:42:53.798479 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.798473    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4e29f82-02fc-42ba-a375-f8d9bc88123c-dshm\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:42:53.798479 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.798482    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4e29f82-02fc-42ba-a375-f8d9bc88123c-tls-certs\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:42:53.886376 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.886281    2571 generic.go:356] "Generic (PLEG): container finished" podID="c4e29f82-02fc-42ba-a375-f8d9bc88123c" containerID="18a89f96ae1df6a672467a5370b1359744a0934e400faf3a9d5b8f498c56e08f" exitCode=0
Sep 15 07:42:53.886376 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.886315    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"
Sep 15 07:42:53.886376 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.886345    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct" event={"ID":"c4e29f82-02fc-42ba-a375-f8d9bc88123c","Type":"ContainerDied","Data":"18a89f96ae1df6a672467a5370b1359744a0934e400faf3a9d5b8f498c56e08f"}
Sep 15 07:42:53.886860 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.886386    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct" event={"ID":"c4e29f82-02fc-42ba-a375-f8d9bc88123c","Type":"ContainerDied","Data":"07f842df6d529d4af6fe6eae4573f6c8bb0f9fee65de75748737c52a56b1338a"}
Sep 15 07:42:53.886860 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.886404    2571 scope.go:117] "RemoveContainer" containerID="18a89f96ae1df6a672467a5370b1359744a0934e400faf3a9d5b8f498c56e08f"
Sep 15 07:42:53.895799 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.895780    2571 scope.go:117] "RemoveContainer" containerID="18a89f96ae1df6a672467a5370b1359744a0934e400faf3a9d5b8f498c56e08f"
Sep 15 07:42:53.896089 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:42:53.896070    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a89f96ae1df6a672467a5370b1359744a0934e400faf3a9d5b8f498c56e08f\": container with ID starting with 18a89f96ae1df6a672467a5370b1359744a0934e400faf3a9d5b8f498c56e08f not found: ID does not exist" containerID="18a89f96ae1df6a672467a5370b1359744a0934e400faf3a9d5b8f498c56e08f"
Sep 15 07:42:53.896137 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.896098    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a89f96ae1df6a672467a5370b1359744a0934e400faf3a9d5b8f498c56e08f"} err="failed to get container status \"18a89f96ae1df6a672467a5370b1359744a0934e400faf3a9d5b8f498c56e08f\": rpc error: code = NotFound desc = could not find container \"18a89f96ae1df6a672467a5370b1359744a0934e400faf3a9d5b8f498c56e08f\": container with ID starting with 18a89f96ae1df6a672467a5370b1359744a0934e400faf3a9d5b8f498c56e08f not found: ID does not exist"
Sep 15 07:42:53.910819 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.910795    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83d162c4-ae04-4957-8775-41dbe5ce03c5" path="/var/lib/kubelet/pods/83d162c4-ae04-4957-8775-41dbe5ce03c5/volumes"
Sep 15 07:42:53.911158 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.911146    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6d1224c-ff36-4220-ab57-63666a719501" path="/var/lib/kubelet/pods/c6d1224c-ff36-4220-ab57-63666a719501/volumes"
Sep 15 07:42:53.911424 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.911411    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"]
Sep 15 07:42:53.911465 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:53.911435    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-fcd453/test-model-fcd453-kserve-c78759c9-96pct"]
Sep 15 07:42:55.910994 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:55.910958    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4e29f82-02fc-42ba-a375-f8d9bc88123c" path="/var/lib/kubelet/pods/c4e29f82-02fc-42ba-a375-f8d9bc88123c/volumes"
Sep 15 07:42:58.821772 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:58.821734    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fcd453-wj8ch"]
Sep 15 07:42:58.822278 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:58.822182    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3f34c4a-b956-4c19-aaee-f85fdef80b99" containerName="main"
Sep 15 07:42:58.822278 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:58.822194    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3f34c4a-b956-4c19-aaee-f85fdef80b99" containerName="main"
Sep 15 07:42:58.822278 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:58.822209    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6d1224c-ff36-4220-ab57-63666a719501" containerName="payload-pre-processing"
Sep 15 07:42:58.822278 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:58.822214    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6d1224c-ff36-4220-ab57-63666a719501" containerName="payload-pre-processing"
Sep 15 07:42:58.822278 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:58.822224    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4e29f82-02fc-42ba-a375-f8d9bc88123c" containerName="main"
Sep 15 07:42:58.822278 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:58.822231    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4e29f82-02fc-42ba-a375-f8d9bc88123c" containerName="main"
Sep 15 07:42:58.822278 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:58.822238    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83d162c4-ae04-4957-8775-41dbe5ce03c5" containerName="revoke-keys"
Sep 15 07:42:58.822278 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:58.822244    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="83d162c4-ae04-4957-8775-41dbe5ce03c5" containerName="revoke-keys"
Sep 15 07:42:58.822535 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:58.822333    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6d1224c-ff36-4220-ab57-63666a719501" containerName="payload-pre-processing"
Sep 15 07:42:58.822535 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:58.822344    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3f34c4a-b956-4c19-aaee-f85fdef80b99" containerName="main"
Sep 15 07:42:58.822535 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:58.822350    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4e29f82-02fc-42ba-a375-f8d9bc88123c" containerName="main"
Sep 15 07:42:58.822535 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:58.822359    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="83d162c4-ae04-4957-8775-41dbe5ce03c5" containerName="revoke-keys"
Sep 15 07:42:58.826606 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:58.826586    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fcd453-wj8ch"
Sep 15 07:42:58.829014 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:58.828986    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8jpwk\""
Sep 15 07:42:58.829121 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:58.828992    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 07:42:58.832623 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:58.832599    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fcd453-wj8ch"]
Sep 15 07:42:58.944415 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:58.944377    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03c8cb9-820d-4f64-89db-4a081c0854bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fcd453-wj8ch\" (UID: \"c03c8cb9-820d-4f64-89db-4a081c0854bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fcd453-wj8ch"
Sep 15 07:42:59.045001 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:59.044964    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03c8cb9-820d-4f64-89db-4a081c0854bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fcd453-wj8ch\" (UID: \"c03c8cb9-820d-4f64-89db-4a081c0854bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fcd453-wj8ch"
Sep 15 07:42:59.045650 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:59.045630    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03c8cb9-820d-4f64-89db-4a081c0854bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fcd453-wj8ch\" (UID: \"c03c8cb9-820d-4f64-89db-4a081c0854bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fcd453-wj8ch"
Sep 15 07:42:59.138172 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:59.138090    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fcd453-wj8ch"
Sep 15 07:42:59.410756 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:59.410728    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fcd453-wj8ch"]
Sep 15 07:42:59.412164 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:42:59.412134    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03c8cb9_820d_4f64_89db_4a081c0854bc.slice/crio-b9081bb2537b126fb721df39a7d712f502782cb0c7d153d0383b0dc490f468e1 WatchSource:0}: Error finding container b9081bb2537b126fb721df39a7d712f502782cb0c7d153d0383b0dc490f468e1: Status 404 returned error can't find the container with id b9081bb2537b126fb721df39a7d712f502782cb0c7d153d0383b0dc490f468e1
Sep 15 07:42:59.921694 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:59.921654    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fcd453-wj8ch" event={"ID":"c03c8cb9-820d-4f64-89db-4a081c0854bc","Type":"ContainerStarted","Data":"1d1157322e45d6c6fc4695abbd4576c8031a62fa9ea3c6d4b45b2f1078531604"}
Sep 15 07:42:59.922075 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:59.921709    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fcd453-wj8ch" event={"ID":"c03c8cb9-820d-4f64-89db-4a081c0854bc","Type":"ContainerStarted","Data":"b9081bb2537b126fb721df39a7d712f502782cb0c7d153d0383b0dc490f468e1"}
Sep 15 07:42:59.936136 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:42:59.936083    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fcd453-wj8ch" podStartSLOduration=1.936067569 podStartE2EDuration="1.936067569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:42:59.935775866 +0000 UTC m=+1484.482345979" watchObservedRunningTime="2026-09-15 07:42:59.936067569 +0000 UTC m=+1484.482637667"
Sep 15 07:43:00.930209 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:00.930161    2571 generic.go:356] "Generic (PLEG): container finished" podID="c03c8cb9-820d-4f64-89db-4a081c0854bc" containerID="1d1157322e45d6c6fc4695abbd4576c8031a62fa9ea3c6d4b45b2f1078531604" exitCode=0
Sep 15 07:43:00.930603 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:00.930270    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fcd453-wj8ch" event={"ID":"c03c8cb9-820d-4f64-89db-4a081c0854bc","Type":"ContainerDied","Data":"1d1157322e45d6c6fc4695abbd4576c8031a62fa9ea3c6d4b45b2f1078531604"}
Sep 15 07:43:01.013173 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.013129    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"]
Sep 15 07:43:01.017422 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.017392    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:01.020366 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.020338    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2735c5\"/\"distinct-2-w0-2735c5-kserve-self-signed-certs\""
Sep 15 07:43:01.026415 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.026392    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"]
Sep 15 07:43:01.066350 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.066318    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-dshm\") pod \"distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:01.066586 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.066560    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4190df-9881-42fc-bda5-b73881600bf6-tls-certs\") pod \"distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:01.066793 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.066771    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-home\") pod \"distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:01.066858 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.066807    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gkg8\" (UniqueName: \"kubernetes.io/projected/6c4190df-9881-42fc-bda5-b73881600bf6-kube-api-access-7gkg8\") pod \"distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:01.066858 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.066849    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-tmp-dir\") pod \"distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:01.066942 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.066874    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-model-cache\") pod \"distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:01.167417 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.167383    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-home\") pod \"distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:01.167417 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.167419    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gkg8\" (UniqueName: \"kubernetes.io/projected/6c4190df-9881-42fc-bda5-b73881600bf6-kube-api-access-7gkg8\") pod \"distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:01.167660 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.167443    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-tmp-dir\") pod \"distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:01.167722 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.167646    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-model-cache\") pod \"distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:01.167722 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.167704    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-dshm\") pod \"distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:01.167821 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.167748    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4190df-9881-42fc-bda5-b73881600bf6-tls-certs\") pod \"distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:01.167821 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.167802    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-tmp-dir\") pod \"distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:01.167989 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.167820    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-home\") pod \"distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:01.167989 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.167923    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-model-cache\") pod \"distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:01.169928 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.169900    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-dshm\") pod \"distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:01.170210 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.170192    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4190df-9881-42fc-bda5-b73881600bf6-tls-certs\") pod \"distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:01.174781 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.174764    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gkg8\" (UniqueName: \"kubernetes.io/projected/6c4190df-9881-42fc-bda5-b73881600bf6-kube-api-access-7gkg8\") pod \"distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:01.328408 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.328363    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:01.603072 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.602997    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"]
Sep 15 07:43:01.606723 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:43:01.606696    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c4190df_9881_42fc_bda5_b73881600bf6.slice/crio-1f88bba3d23f0307ea08c1f2180c896f8a47da9b2d923e46d69297d17cd4f6b2 WatchSource:0}: Error finding container 1f88bba3d23f0307ea08c1f2180c896f8a47da9b2d923e46d69297d17cd4f6b2: Status 404 returned error can't find the container with id 1f88bba3d23f0307ea08c1f2180c896f8a47da9b2d923e46d69297d17cd4f6b2
Sep 15 07:43:01.938448 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.938351    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp" event={"ID":"6c4190df-9881-42fc-bda5-b73881600bf6","Type":"ContainerStarted","Data":"cf3f6c8128999ad212b987fd45e888de8f8d61e161e971ebdef341af564d40f6"}
Sep 15 07:43:01.938448 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.938401    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp" event={"ID":"6c4190df-9881-42fc-bda5-b73881600bf6","Type":"ContainerStarted","Data":"1f88bba3d23f0307ea08c1f2180c896f8a47da9b2d923e46d69297d17cd4f6b2"}
Sep 15 07:43:01.955630 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:01.955568    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp" podStartSLOduration=1.955545128 podStartE2EDuration="1.955545128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:43:01.955379251 +0000 UTC m=+1486.501949353" watchObservedRunningTime="2026-09-15 07:43:01.955545128 +0000 UTC m=+1486.502115228"
Sep 15 07:43:02.063294 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:02.063269    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fcd453-wj8ch"
Sep 15 07:43:02.178948 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:02.178914    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03c8cb9-820d-4f64-89db-4a081c0854bc-maas-api-service-ca\") pod \"c03c8cb9-820d-4f64-89db-4a081c0854bc\" (UID: \"c03c8cb9-820d-4f64-89db-4a081c0854bc\") "
Sep 15 07:43:02.179334 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:02.179309    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c03c8cb9-820d-4f64-89db-4a081c0854bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c03c8cb9-820d-4f64-89db-4a081c0854bc" (UID: "c03c8cb9-820d-4f64-89db-4a081c0854bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:43:02.279729 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:02.279694    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03c8cb9-820d-4f64-89db-4a081c0854bc-maas-api-service-ca\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:43:02.946243 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:02.946213    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fcd453-wj8ch"
Sep 15 07:43:02.946643 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:02.946277    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fcd453-wj8ch" event={"ID":"c03c8cb9-820d-4f64-89db-4a081c0854bc","Type":"ContainerDied","Data":"b9081bb2537b126fb721df39a7d712f502782cb0c7d153d0383b0dc490f468e1"}
Sep 15 07:43:02.946643 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:02.946339    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9081bb2537b126fb721df39a7d712f502782cb0c7d153d0383b0dc490f468e1"
Sep 15 07:43:09.350345 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:09.350306    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx"]
Sep 15 07:43:09.350840 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:09.350571    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx" podUID="95397f4f-11d5-4393-94ce-3d29ea930c5a" containerName="payload-processing" containerID="cri-o://7e1b84b69213bc20e4ac29b5808b742c032e3a669e2e96ced6ad542e154d5a9a" gracePeriod=30
Sep 15 07:43:09.531491 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:09.531452    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fcd453-wj8ch"]
Sep 15 07:43:09.533803 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:09.533777    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fcd453-wj8ch"]
Sep 15 07:43:09.604054 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:09.603981    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx"
Sep 15 07:43:09.754381 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:09.754222    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvnvp\" (UniqueName: \"kubernetes.io/projected/95397f4f-11d5-4393-94ce-3d29ea930c5a-kube-api-access-bvnvp\") pod \"95397f4f-11d5-4393-94ce-3d29ea930c5a\" (UID: \"95397f4f-11d5-4393-94ce-3d29ea930c5a\") "
Sep 15 07:43:09.754381 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:09.754324    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95397f4f-11d5-4393-94ce-3d29ea930c5a-plugins-config-volume\") pod \"95397f4f-11d5-4393-94ce-3d29ea930c5a\" (UID: \"95397f4f-11d5-4393-94ce-3d29ea930c5a\") "
Sep 15 07:43:09.754767 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:09.754736    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95397f4f-11d5-4393-94ce-3d29ea930c5a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "95397f4f-11d5-4393-94ce-3d29ea930c5a" (UID: "95397f4f-11d5-4393-94ce-3d29ea930c5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:43:09.756440 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:09.756415    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95397f4f-11d5-4393-94ce-3d29ea930c5a-kube-api-access-bvnvp" (OuterVolumeSpecName: "kube-api-access-bvnvp") pod "95397f4f-11d5-4393-94ce-3d29ea930c5a" (UID: "95397f4f-11d5-4393-94ce-3d29ea930c5a"). InnerVolumeSpecName "kube-api-access-bvnvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:43:09.856315 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:09.856214    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvnvp\" (UniqueName: \"kubernetes.io/projected/95397f4f-11d5-4393-94ce-3d29ea930c5a-kube-api-access-bvnvp\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:43:09.856315 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:09.856247    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95397f4f-11d5-4393-94ce-3d29ea930c5a-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:43:09.909996 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:09.909961    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c03c8cb9-820d-4f64-89db-4a081c0854bc" path="/var/lib/kubelet/pods/c03c8cb9-820d-4f64-89db-4a081c0854bc/volumes"
Sep 15 07:43:09.990767 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:09.990731    2571 generic.go:356] "Generic (PLEG): container finished" podID="95397f4f-11d5-4393-94ce-3d29ea930c5a" containerID="7e1b84b69213bc20e4ac29b5808b742c032e3a669e2e96ced6ad542e154d5a9a" exitCode=0
Sep 15 07:43:09.990937 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:09.990772    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx"
Sep 15 07:43:09.990937 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:09.990811    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx" event={"ID":"95397f4f-11d5-4393-94ce-3d29ea930c5a","Type":"ContainerDied","Data":"7e1b84b69213bc20e4ac29b5808b742c032e3a669e2e96ced6ad542e154d5a9a"}
Sep 15 07:43:09.990937 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:09.990865    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx" event={"ID":"95397f4f-11d5-4393-94ce-3d29ea930c5a","Type":"ContainerDied","Data":"ee90f670116b45a57148a0574a7a3dafa9bf57d64aefea801187b560122ef2e8"}
Sep 15 07:43:09.990937 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:09.990888    2571 scope.go:117] "RemoveContainer" containerID="7e1b84b69213bc20e4ac29b5808b742c032e3a669e2e96ced6ad542e154d5a9a"
Sep 15 07:43:10.000484 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:10.000461    2571 scope.go:117] "RemoveContainer" containerID="7e1b84b69213bc20e4ac29b5808b742c032e3a669e2e96ced6ad542e154d5a9a"
Sep 15 07:43:10.000745 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:43:10.000724    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e1b84b69213bc20e4ac29b5808b742c032e3a669e2e96ced6ad542e154d5a9a\": container with ID starting with 7e1b84b69213bc20e4ac29b5808b742c032e3a669e2e96ced6ad542e154d5a9a not found: ID does not exist" containerID="7e1b84b69213bc20e4ac29b5808b742c032e3a669e2e96ced6ad542e154d5a9a"
Sep 15 07:43:10.000813 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:10.000759    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e1b84b69213bc20e4ac29b5808b742c032e3a669e2e96ced6ad542e154d5a9a"} err="failed to get container status \"7e1b84b69213bc20e4ac29b5808b742c032e3a669e2e96ced6ad542e154d5a9a\": rpc error: code = NotFound desc = could not find container \"7e1b84b69213bc20e4ac29b5808b742c032e3a669e2e96ced6ad542e154d5a9a\": container with ID starting with 7e1b84b69213bc20e4ac29b5808b742c032e3a669e2e96ced6ad542e154d5a9a not found: ID does not exist"
Sep 15 07:43:10.008562 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:10.008536    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx"]
Sep 15 07:43:10.012770 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:10.012748    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fcd453-d9f4b669c-gv2nx"]
Sep 15 07:43:11.329243 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:11.329205    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:11.329681 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:11.329360    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:11.341853 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:11.341827    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:11.910465 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:11.910431    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95397f4f-11d5-4393-94ce-3d29ea930c5a" path="/var/lib/kubelet/pods/95397f4f-11d5-4393-94ce-3d29ea930c5a/volumes"
Sep 15 07:43:12.015952 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:12.015924    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:43:25.238135 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:25.238094    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz"]
Sep 15 07:43:25.238748 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:25.238726    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95397f4f-11d5-4393-94ce-3d29ea930c5a" containerName="payload-processing"
Sep 15 07:43:25.238835 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:25.238750    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="95397f4f-11d5-4393-94ce-3d29ea930c5a" containerName="payload-processing"
Sep 15 07:43:25.238835 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:25.238767    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c03c8cb9-820d-4f64-89db-4a081c0854bc" containerName="revoke-keys"
Sep 15 07:43:25.238835 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:25.238776    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c03c8cb9-820d-4f64-89db-4a081c0854bc" containerName="revoke-keys"
Sep 15 07:43:25.238980 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:25.238916    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="95397f4f-11d5-4393-94ce-3d29ea930c5a" containerName="payload-processing"
Sep 15 07:43:25.238980 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:25.238935    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c03c8cb9-820d-4f64-89db-4a081c0854bc" containerName="revoke-keys"
Sep 15 07:43:25.242013 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:25.241992    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz"
Sep 15 07:43:25.244353 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:25.244334    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-04464b-dockercfg-ndsf9\""
Sep 15 07:43:25.244593 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:25.244577    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-04464b\""
Sep 15 07:43:25.250583 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:25.250560    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz"]
Sep 15 07:43:25.307179 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:25.307145    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f79da6-cffd-4b19-8e53-da4eb7eaca81-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz\" (UID: \"e4f79da6-cffd-4b19-8e53-da4eb7eaca81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz"
Sep 15 07:43:25.307179 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:25.307182    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqs46\" (UniqueName: \"kubernetes.io/projected/e4f79da6-cffd-4b19-8e53-da4eb7eaca81-kube-api-access-dqs46\") pod \"payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz\" (UID: \"e4f79da6-cffd-4b19-8e53-da4eb7eaca81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz"
Sep 15 07:43:25.408541 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:25.408502    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqs46\" (UniqueName: \"kubernetes.io/projected/e4f79da6-cffd-4b19-8e53-da4eb7eaca81-kube-api-access-dqs46\") pod \"payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz\" (UID: \"e4f79da6-cffd-4b19-8e53-da4eb7eaca81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz"
Sep 15 07:43:25.408711 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:25.408667    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f79da6-cffd-4b19-8e53-da4eb7eaca81-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz\" (UID: \"e4f79da6-cffd-4b19-8e53-da4eb7eaca81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz"
Sep 15 07:43:25.409351 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:25.409327    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f79da6-cffd-4b19-8e53-da4eb7eaca81-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz\" (UID: \"e4f79da6-cffd-4b19-8e53-da4eb7eaca81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz"
Sep 15 07:43:25.416658 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:25.416639    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqs46\" (UniqueName: \"kubernetes.io/projected/e4f79da6-cffd-4b19-8e53-da4eb7eaca81-kube-api-access-dqs46\") pod \"payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz\" (UID: \"e4f79da6-cffd-4b19-8e53-da4eb7eaca81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz"
Sep 15 07:43:25.552755 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:25.552674    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz"
Sep 15 07:43:25.828740 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:25.828715    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz"]
Sep 15 07:43:25.830392 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:43:25.830365    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f79da6_cffd_4b19_8e53_da4eb7eaca81.slice/crio-88e3e38822ed4f3695c274263388ef406b9b9f0e2d3c6ae5d0a2a9dd44c4839f WatchSource:0}: Error finding container 88e3e38822ed4f3695c274263388ef406b9b9f0e2d3c6ae5d0a2a9dd44c4839f: Status 404 returned error can't find the container with id 88e3e38822ed4f3695c274263388ef406b9b9f0e2d3c6ae5d0a2a9dd44c4839f
Sep 15 07:43:26.092418 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:26.092314    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz" event={"ID":"e4f79da6-cffd-4b19-8e53-da4eb7eaca81","Type":"ContainerStarted","Data":"16569fb0034aa0d6f35bbe93679f39c35b25b1475c2c460bb4216a243b9e6912"}
Sep 15 07:43:26.092418 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:26.092388    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz"
Sep 15 07:43:26.092418 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:26.092407    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz" event={"ID":"e4f79da6-cffd-4b19-8e53-da4eb7eaca81","Type":"ContainerStarted","Data":"88e3e38822ed4f3695c274263388ef406b9b9f0e2d3c6ae5d0a2a9dd44c4839f"}
Sep 15 07:43:26.109076 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:26.109024    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz" podStartSLOduration=1.1090067160000001 podStartE2EDuration="1.109006716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:43:26.107994408 +0000 UTC m=+1510.654564526" watchObservedRunningTime="2026-09-15 07:43:26.109006716 +0000 UTC m=+1510.655576855"
Sep 15 07:43:37.100954 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:37.100921    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz"
Sep 15 07:43:50.505761 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.505729    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"]
Sep 15 07:43:50.509392 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.509372    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:43:50.511655 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.511635    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f56165\"/\"e2e-conflict-b-30d020fe-kserve-self-signed-certs\""
Sep 15 07:43:50.519950 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.519928    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"]
Sep 15 07:43:50.566841 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.566799    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-home\") pod \"e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:43:50.566841 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.566843    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbx4s\" (UniqueName: \"kubernetes.io/projected/f6be841e-c24f-4b66-ad25-4528713aef88-kube-api-access-mbx4s\") pod \"e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:43:50.567057 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.566867    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-tmp-dir\") pod \"e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:43:50.567057 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.566895    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6be841e-c24f-4b66-ad25-4528713aef88-tls-certs\") pod \"e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:43:50.567057 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.566998    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-dshm\") pod \"e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:43:50.567057 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.567047    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-model-cache\") pod \"e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:43:50.667755 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.667711    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-dshm\") pod \"e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:43:50.667919 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.667783    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-model-cache\") pod \"e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:43:50.667919 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.667824    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-home\") pod \"e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:43:50.667919 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.667855    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbx4s\" (UniqueName: \"kubernetes.io/projected/f6be841e-c24f-4b66-ad25-4528713aef88-kube-api-access-mbx4s\") pod \"e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:43:50.668083 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.667982    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-tmp-dir\") pod \"e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:43:50.668083 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.668062    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6be841e-c24f-4b66-ad25-4528713aef88-tls-certs\") pod \"e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:43:50.668231 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.668206    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-model-cache\") pod \"e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:43:50.668356 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.668244    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-home\") pod \"e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:43:50.668356 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.668341    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-tmp-dir\") pod \"e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:43:50.669958 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.669930    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-dshm\") pod \"e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:43:50.670412 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.670396    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6be841e-c24f-4b66-ad25-4528713aef88-tls-certs\") pod \"e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:43:50.675438 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.675415    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbx4s\" (UniqueName: \"kubernetes.io/projected/f6be841e-c24f-4b66-ad25-4528713aef88-kube-api-access-mbx4s\") pod \"e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") " pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:43:50.820310 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:50.820204    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:43:51.109221 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:51.109193    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"]
Sep 15 07:43:51.111664 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:43:51.111635    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6be841e_c24f_4b66_ad25_4528713aef88.slice/crio-26b9641de28aeda2af2c62142c9104c850d820a6979a6f89a8321bc4d6b2f8a7 WatchSource:0}: Error finding container 26b9641de28aeda2af2c62142c9104c850d820a6979a6f89a8321bc4d6b2f8a7: Status 404 returned error can't find the container with id 26b9641de28aeda2af2c62142c9104c850d820a6979a6f89a8321bc4d6b2f8a7
Sep 15 07:43:51.230233 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:51.230192    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg" event={"ID":"f6be841e-c24f-4b66-ad25-4528713aef88","Type":"ContainerStarted","Data":"ef94bb7760fc3bc9868071df0a743deb7c2c736e811248c73bf94f11a4e547d0"}
Sep 15 07:43:51.230412 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:51.230244    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg" event={"ID":"f6be841e-c24f-4b66-ad25-4528713aef88","Type":"ContainerStarted","Data":"26b9641de28aeda2af2c62142c9104c850d820a6979a6f89a8321bc4d6b2f8a7"}
Sep 15 07:43:51.247896 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:51.247847    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg" podStartSLOduration=1.247831481 podStartE2EDuration="1.247831481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:43:51.246212227 +0000 UTC m=+1535.792782351" watchObservedRunningTime="2026-09-15 07:43:51.247831481 +0000 UTC m=+1535.794401579"
Sep 15 07:43:59.652549 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:59.652505    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p"]
Sep 15 07:43:59.656394 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:59.656372    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p"
Sep 15 07:43:59.659101 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:59.659074    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-e3a482-dockercfg-jmlpd\""
Sep 15 07:43:59.659214 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:59.659076    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-e3a482\""
Sep 15 07:43:59.666685 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:59.666654    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p"]
Sep 15 07:43:59.750249 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:59.750209    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t56q5\" (UniqueName: \"kubernetes.io/projected/45fcb589-c2be-4abf-b417-293704470729-kube-api-access-t56q5\") pod \"payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p\" (UID: \"45fcb589-c2be-4abf-b417-293704470729\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p"
Sep 15 07:43:59.750424 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:59.750406    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45fcb589-c2be-4abf-b417-293704470729-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p\" (UID: \"45fcb589-c2be-4abf-b417-293704470729\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p"
Sep 15 07:43:59.851438 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:59.851399    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45fcb589-c2be-4abf-b417-293704470729-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p\" (UID: \"45fcb589-c2be-4abf-b417-293704470729\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p"
Sep 15 07:43:59.851608 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:59.851484    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t56q5\" (UniqueName: \"kubernetes.io/projected/45fcb589-c2be-4abf-b417-293704470729-kube-api-access-t56q5\") pod \"payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p\" (UID: \"45fcb589-c2be-4abf-b417-293704470729\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p"
Sep 15 07:43:59.852065 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:59.852044    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45fcb589-c2be-4abf-b417-293704470729-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p\" (UID: \"45fcb589-c2be-4abf-b417-293704470729\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p"
Sep 15 07:43:59.859467 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:59.859440    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t56q5\" (UniqueName: \"kubernetes.io/projected/45fcb589-c2be-4abf-b417-293704470729-kube-api-access-t56q5\") pod \"payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p\" (UID: \"45fcb589-c2be-4abf-b417-293704470729\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p"
Sep 15 07:43:59.968228 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:43:59.968195    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p"
Sep 15 07:44:00.238149 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:44:00.238124    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p"]
Sep 15 07:44:00.239692 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:44:00.239665    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45fcb589_c2be_4abf_b417_293704470729.slice/crio-dc951d2ce40557f6a415a2a49cc6f45d0d5621ec0b2622a687601cef26cf839e WatchSource:0}: Error finding container dc951d2ce40557f6a415a2a49cc6f45d0d5621ec0b2622a687601cef26cf839e: Status 404 returned error can't find the container with id dc951d2ce40557f6a415a2a49cc6f45d0d5621ec0b2622a687601cef26cf839e
Sep 15 07:44:00.281611 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:44:00.281579    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p" event={"ID":"45fcb589-c2be-4abf-b417-293704470729","Type":"ContainerStarted","Data":"dc951d2ce40557f6a415a2a49cc6f45d0d5621ec0b2622a687601cef26cf839e"}
Sep 15 07:44:00.820327 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:44:00.820297    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:44:00.820327 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:44:00.820336    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:44:00.833171 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:44:00.833142    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:44:01.289089 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:44:01.289054    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p" event={"ID":"45fcb589-c2be-4abf-b417-293704470729","Type":"ContainerStarted","Data":"cec86f353ddde39d25b652f4f5a11cd15d0c16980c8d285760c03f8a94d5a54a"}
Sep 15 07:44:01.289369 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:44:01.289350    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p"
Sep 15 07:44:01.300313 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:44:01.300288    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:44:01.304570 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:44:01.304522    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p" podStartSLOduration=2.304505564 podStartE2EDuration="2.304505564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:44:01.304056985 +0000 UTC m=+1545.850627103" watchObservedRunningTime="2026-09-15 07:44:01.304505564 +0000 UTC m=+1545.851075665"
Sep 15 07:44:12.296564 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:44:12.296531    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p"
Sep 15 07:44:16.024895 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:44:16.024856    2571 scope.go:117] "RemoveContainer" containerID="6a395779463aa62e5ebce2002e6d831d9cc741160f1b45f48b6440cf56911bb0"
Sep 15 07:44:16.033527 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:44:16.033504    2571 scope.go:117] "RemoveContainer" containerID="7cfb37b15f2e8fe4c475155304af524018bc6a8a6d6fecdbd4742be6f5e01509"
Sep 15 07:45:00.142120 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.142079    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2735c5-29824305-wfzgm"]
Sep 15 07:45:00.146497 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.146477    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3a4c9f-29824305-q84rz"]
Sep 15 07:45:00.146604 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.146583    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2735c5-29824305-wfzgm"
Sep 15 07:45:00.148932 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.148901    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8jpwk\""
Sep 15 07:45:00.150111 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.150095    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824305-jjpqg"]
Sep 15 07:45:00.150204 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.150192    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3a4c9f-29824305-q84rz"
Sep 15 07:45:00.153564 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.153539    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824305-jjpqg"
Sep 15 07:45:00.154489 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.154431    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3a4c9f-29824305-q84rz"]
Sep 15 07:45:00.160299 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.160276    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2735c5-29824305-wfzgm"
Sep 15 07:45:00.161763 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.161746    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824305-jjpqg"]
Sep 15 07:45:00.165150 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.164464    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2735c5-29824305-wfzgm"]
Sep 15 07:45:00.171911 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.171886    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3a4c9f-29824305-q84rz"
Sep 15 07:45:00.174347 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.174323    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-04464b-29824305-qxv5t"]
Sep 15 07:45:00.178226 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.178201    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-04464b-29824305-qxv5t"
Sep 15 07:45:00.188801 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.188372    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-04464b-29824305-qxv5t"]
Sep 15 07:45:00.221894 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.221865    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824305-jjpqg"
Sep 15 07:45:00.234616 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.234591    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-04464b-29824305-qxv5t"
Sep 15 07:45:00.448415 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.448387    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2735c5-29824305-wfzgm"]
Sep 15 07:45:00.449684 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:45:00.449659    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace9f8ee_1513_483d_b257_5c3f830c66b4.slice/crio-78231a04d98a25c8f4f88d83c2a2fc2484f566cd3a3a2167cb280ac845358acc WatchSource:0}: Error finding container 78231a04d98a25c8f4f88d83c2a2fc2484f566cd3a3a2167cb280ac845358acc: Status 404 returned error can't find the container with id 78231a04d98a25c8f4f88d83c2a2fc2484f566cd3a3a2167cb280ac845358acc
Sep 15 07:45:00.472475 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.472451    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3a4c9f-29824305-q84rz"]
Sep 15 07:45:00.485432 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:45:00.485406    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4810f8_42e1_438d_94f1_83ebb42caaa7.slice/crio-99086fcbe4d2098a3ada9991eec5b7bdf18f8ec6a34c1bc789d00cf2d3fdfd17 WatchSource:0}: Error finding container 99086fcbe4d2098a3ada9991eec5b7bdf18f8ec6a34c1bc789d00cf2d3fdfd17: Status 404 returned error can't find the container with id 99086fcbe4d2098a3ada9991eec5b7bdf18f8ec6a34c1bc789d00cf2d3fdfd17
Sep 15 07:45:00.529843 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.529816    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824305-jjpqg"]
Sep 15 07:45:00.532042 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:45:00.532009    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8070eeec_fbdd_42d5_8c06_a195b2a25eb4.slice/crio-1b9bf0ebd1c6b1f99effc0f612a5c0a88cf19108c189f1fb20d0806d4dffc4f8 WatchSource:0}: Error finding container 1b9bf0ebd1c6b1f99effc0f612a5c0a88cf19108c189f1fb20d0806d4dffc4f8: Status 404 returned error can't find the container with id 1b9bf0ebd1c6b1f99effc0f612a5c0a88cf19108c189f1fb20d0806d4dffc4f8
Sep 15 07:45:00.532650 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.532622    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-04464b-29824305-qxv5t"]
Sep 15 07:45:00.533156 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:45:00.533134    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9dd07f_838f_4c4e_9167_cb4cc46e1731.slice/crio-88bea0e160b3972929766a8d4050f870e4115c1a5d0d2a7a1e2c166c7956a6cc WatchSource:0}: Error finding container 88bea0e160b3972929766a8d4050f870e4115c1a5d0d2a7a1e2c166c7956a6cc: Status 404 returned error can't find the container with id 88bea0e160b3972929766a8d4050f870e4115c1a5d0d2a7a1e2c166c7956a6cc
Sep 15 07:45:00.617662 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.617628    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3a4c9f-29824305-q84rz" event={"ID":"0d4810f8-42e1-438d-94f1-83ebb42caaa7","Type":"ContainerStarted","Data":"99086fcbe4d2098a3ada9991eec5b7bdf18f8ec6a34c1bc789d00cf2d3fdfd17"}
Sep 15 07:45:00.618865 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.618838    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2735c5-29824305-wfzgm" event={"ID":"ace9f8ee-1513-483d-b257-5c3f830c66b4","Type":"ContainerStarted","Data":"78231a04d98a25c8f4f88d83c2a2fc2484f566cd3a3a2167cb280ac845358acc"}
Sep 15 07:45:00.620015 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.619991    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-04464b-29824305-qxv5t" event={"ID":"8070eeec-fbdd-42d5-8c06-a195b2a25eb4","Type":"ContainerStarted","Data":"1b9bf0ebd1c6b1f99effc0f612a5c0a88cf19108c189f1fb20d0806d4dffc4f8"}
Sep 15 07:45:00.621083 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:00.621065    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824305-jjpqg" event={"ID":"bc9dd07f-838f-4c4e-9167-cb4cc46e1731","Type":"ContainerStarted","Data":"88bea0e160b3972929766a8d4050f870e4115c1a5d0d2a7a1e2c166c7956a6cc"}
Sep 15 07:45:01.627662 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:01.627625    2571 generic.go:356] "Generic (PLEG): container finished" podID="0d4810f8-42e1-438d-94f1-83ebb42caaa7" containerID="cd64ba6ac169ac0276996430de60cd3967705a72dff2b0de2c4651935296e934" exitCode=0
Sep 15 07:45:01.628025 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:01.627728    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3a4c9f-29824305-q84rz" event={"ID":"0d4810f8-42e1-438d-94f1-83ebb42caaa7","Type":"ContainerDied","Data":"cd64ba6ac169ac0276996430de60cd3967705a72dff2b0de2c4651935296e934"}
Sep 15 07:45:01.629249 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:01.629228    2571 generic.go:356] "Generic (PLEG): container finished" podID="ace9f8ee-1513-483d-b257-5c3f830c66b4" containerID="610c016083d20f4bd4693c5c0ef6c882e5d2dfcea64c802f213afd40b8514aba" exitCode=0
Sep 15 07:45:01.629372 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:01.629302    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2735c5-29824305-wfzgm" event={"ID":"ace9f8ee-1513-483d-b257-5c3f830c66b4","Type":"ContainerDied","Data":"610c016083d20f4bd4693c5c0ef6c882e5d2dfcea64c802f213afd40b8514aba"}
Sep 15 07:45:01.630811 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:01.630790    2571 generic.go:356] "Generic (PLEG): container finished" podID="8070eeec-fbdd-42d5-8c06-a195b2a25eb4" containerID="42809aa23832d8b3ac3fc69fcd3c3144de016ca93de8d8f6cd7ec71c65c6407e" exitCode=0
Sep 15 07:45:01.630888 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:01.630852    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-04464b-29824305-qxv5t" event={"ID":"8070eeec-fbdd-42d5-8c06-a195b2a25eb4","Type":"ContainerDied","Data":"42809aa23832d8b3ac3fc69fcd3c3144de016ca93de8d8f6cd7ec71c65c6407e"}
Sep 15 07:45:01.632332 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:01.632302    2571 generic.go:356] "Generic (PLEG): container finished" podID="bc9dd07f-838f-4c4e-9167-cb4cc46e1731" containerID="18079791159d8afd74830e533523589aa06022d849ec2a8ca91715c8b05f365c" exitCode=0
Sep 15 07:45:01.632405 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:01.632383    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824305-jjpqg" event={"ID":"bc9dd07f-838f-4c4e-9167-cb4cc46e1731","Type":"ContainerDied","Data":"18079791159d8afd74830e533523589aa06022d849ec2a8ca91715c8b05f365c"}
Sep 15 07:45:02.784201 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:02.784169    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824305-jjpqg"
Sep 15 07:45:02.826508 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:02.826479    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2735c5-29824305-wfzgm"
Sep 15 07:45:02.870453 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:02.870428    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3a4c9f-29824305-q84rz"
Sep 15 07:45:02.873622 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:02.873600    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-04464b-29824305-qxv5t"
Sep 15 07:45:03.646049 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:03.646022    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-04464b-29824305-qxv5t"
Sep 15 07:45:03.646242 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:03.646067    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-04464b-29824305-qxv5t" event={"ID":"8070eeec-fbdd-42d5-8c06-a195b2a25eb4","Type":"ContainerDied","Data":"1b9bf0ebd1c6b1f99effc0f612a5c0a88cf19108c189f1fb20d0806d4dffc4f8"}
Sep 15 07:45:03.646242 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:03.646119    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b9bf0ebd1c6b1f99effc0f612a5c0a88cf19108c189f1fb20d0806d4dffc4f8"
Sep 15 07:45:03.647450 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:03.647427    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824305-jjpqg"
Sep 15 07:45:03.647568 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:03.647466    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824305-jjpqg" event={"ID":"bc9dd07f-838f-4c4e-9167-cb4cc46e1731","Type":"ContainerDied","Data":"88bea0e160b3972929766a8d4050f870e4115c1a5d0d2a7a1e2c166c7956a6cc"}
Sep 15 07:45:03.647568 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:03.647514    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88bea0e160b3972929766a8d4050f870e4115c1a5d0d2a7a1e2c166c7956a6cc"
Sep 15 07:45:03.648942 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:03.648922    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3a4c9f-29824305-q84rz"
Sep 15 07:45:03.648942 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:03.648922    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3a4c9f-29824305-q84rz" event={"ID":"0d4810f8-42e1-438d-94f1-83ebb42caaa7","Type":"ContainerDied","Data":"99086fcbe4d2098a3ada9991eec5b7bdf18f8ec6a34c1bc789d00cf2d3fdfd17"}
Sep 15 07:45:03.649086 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:03.648973    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99086fcbe4d2098a3ada9991eec5b7bdf18f8ec6a34c1bc789d00cf2d3fdfd17"
Sep 15 07:45:03.650409 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:03.650391    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2735c5-29824305-wfzgm" event={"ID":"ace9f8ee-1513-483d-b257-5c3f830c66b4","Type":"ContainerDied","Data":"78231a04d98a25c8f4f88d83c2a2fc2484f566cd3a3a2167cb280ac845358acc"}
Sep 15 07:45:03.650494 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:03.650428    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78231a04d98a25c8f4f88d83c2a2fc2484f566cd3a3a2167cb280ac845358acc"
Sep 15 07:45:03.650494 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:03.650438    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2735c5-29824305-wfzgm"
Sep 15 07:45:59.246635 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.246597    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b21d1ff2-7ccvz"]
Sep 15 07:45:59.247070 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.247057    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ace9f8ee-1513-483d-b257-5c3f830c66b4" containerName="cleanup"
Sep 15 07:45:59.247070 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.247071    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ace9f8ee-1513-483d-b257-5c3f830c66b4" containerName="cleanup"
Sep 15 07:45:59.247147 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.247078    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d4810f8-42e1-438d-94f1-83ebb42caaa7" containerName="cleanup"
Sep 15 07:45:59.247147 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.247084    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d4810f8-42e1-438d-94f1-83ebb42caaa7" containerName="cleanup"
Sep 15 07:45:59.247147 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.247102    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8070eeec-fbdd-42d5-8c06-a195b2a25eb4" containerName="cleanup"
Sep 15 07:45:59.247147 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.247108    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8070eeec-fbdd-42d5-8c06-a195b2a25eb4" containerName="cleanup"
Sep 15 07:45:59.247147 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.247116    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc9dd07f-838f-4c4e-9167-cb4cc46e1731" containerName="cleanup"
Sep 15 07:45:59.247147 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.247121    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc9dd07f-838f-4c4e-9167-cb4cc46e1731" containerName="cleanup"
Sep 15 07:45:59.247361 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.247190    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ace9f8ee-1513-483d-b257-5c3f830c66b4" containerName="cleanup"
Sep 15 07:45:59.247361 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.247199    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8070eeec-fbdd-42d5-8c06-a195b2a25eb4" containerName="cleanup"
Sep 15 07:45:59.247361 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.247205    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc9dd07f-838f-4c4e-9167-cb4cc46e1731" containerName="cleanup"
Sep 15 07:45:59.247361 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.247211    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d4810f8-42e1-438d-94f1-83ebb42caaa7" containerName="cleanup"
Sep 15 07:45:59.250536 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.250515    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b21d1ff2-7ccvz"
Sep 15 07:45:59.252807 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.252782    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 07:45:59.252929 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.252785    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8jpwk\""
Sep 15 07:45:59.257203 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.257174    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b21d1ff2-7ccvz"]
Sep 15 07:45:59.355158 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.355117    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26856b1-41e0-408b-96f3-3148ebdfd757-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b21d1ff2-7ccvz\" (UID: \"c26856b1-41e0-408b-96f3-3148ebdfd757\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b21d1ff2-7ccvz"
Sep 15 07:45:59.456637 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.456600    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26856b1-41e0-408b-96f3-3148ebdfd757-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b21d1ff2-7ccvz\" (UID: \"c26856b1-41e0-408b-96f3-3148ebdfd757\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b21d1ff2-7ccvz"
Sep 15 07:45:59.457294 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.457247    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26856b1-41e0-408b-96f3-3148ebdfd757-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b21d1ff2-7ccvz\" (UID: \"c26856b1-41e0-408b-96f3-3148ebdfd757\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b21d1ff2-7ccvz"
Sep 15 07:45:59.562006 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.561928    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b21d1ff2-7ccvz"
Sep 15 07:45:59.835058 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.835030    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b21d1ff2-7ccvz"]
Sep 15 07:45:59.836405 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:45:59.836374    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26856b1_41e0_408b_96f3_3148ebdfd757.slice/crio-734fda5853915a5f93503c4001305406cc97c9287a41ddac861e5b867ddaf7f5 WatchSource:0}: Error finding container 734fda5853915a5f93503c4001305406cc97c9287a41ddac861e5b867ddaf7f5: Status 404 returned error can't find the container with id 734fda5853915a5f93503c4001305406cc97c9287a41ddac861e5b867ddaf7f5
Sep 15 07:45:59.975284 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:45:59.975228    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b21d1ff2-7ccvz" event={"ID":"c26856b1-41e0-408b-96f3-3148ebdfd757","Type":"ContainerStarted","Data":"734fda5853915a5f93503c4001305406cc97c9287a41ddac861e5b867ddaf7f5"}
Sep 15 07:46:00.984066 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:00.984030    2571 generic.go:356] "Generic (PLEG): container finished" podID="c26856b1-41e0-408b-96f3-3148ebdfd757" containerID="769b98712ac3d59f124aeef251d478f15b8f1933ec3cb7f54bd43fa07c631acc" exitCode=0
Sep 15 07:46:00.984536 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:00.984072    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b21d1ff2-7ccvz" event={"ID":"c26856b1-41e0-408b-96f3-3148ebdfd757","Type":"ContainerDied","Data":"769b98712ac3d59f124aeef251d478f15b8f1933ec3cb7f54bd43fa07c631acc"}
Sep 15 07:46:02.126195 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.126166    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b21d1ff2-7ccvz"
Sep 15 07:46:02.184928 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.184885    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26856b1-41e0-408b-96f3-3148ebdfd757-maas-api-service-ca\") pod \"c26856b1-41e0-408b-96f3-3148ebdfd757\" (UID: \"c26856b1-41e0-408b-96f3-3148ebdfd757\") "
Sep 15 07:46:02.185234 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.185208    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26856b1-41e0-408b-96f3-3148ebdfd757-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c26856b1-41e0-408b-96f3-3148ebdfd757" (UID: "c26856b1-41e0-408b-96f3-3148ebdfd757"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:46:02.285784 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.285707    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26856b1-41e0-408b-96f3-3148ebdfd757-maas-api-service-ca\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:46:02.901754 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.901718    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"]
Sep 15 07:46:02.902279 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.902207    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c26856b1-41e0-408b-96f3-3148ebdfd757" containerName="revoke-keys"
Sep 15 07:46:02.902279 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.902222    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c26856b1-41e0-408b-96f3-3148ebdfd757" containerName="revoke-keys"
Sep 15 07:46:02.902456 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.902347    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c26856b1-41e0-408b-96f3-3148ebdfd757" containerName="revoke-keys"
Sep 15 07:46:02.905659 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.905637    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:02.915161 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.915138    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-04464b\"/\"openshift-service-ca.crt\""
Sep 15 07:46:02.915412 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.915390    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-04464b\"/\"auth-test-model-04464b-kserve-self-signed-certs\""
Sep 15 07:46:02.915412 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.915398    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-04464b\"/\"kube-root-ca.crt\""
Sep 15 07:46:02.915568 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.915403    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-04464b\"/\"default-dockercfg-79sv4\""
Sep 15 07:46:02.948591 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.948558    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"]
Sep 15 07:46:02.992846 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.992812    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksrht\" (UniqueName: \"kubernetes.io/projected/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-kube-api-access-ksrht\") pod \"auth-test-model-04464b-kserve-6ddb8bb859-m7gk4\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") " pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:02.993037 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.992876    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-home\") pod \"auth-test-model-04464b-kserve-6ddb8bb859-m7gk4\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") " pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:02.993037 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.992935    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-dshm\") pod \"auth-test-model-04464b-kserve-6ddb8bb859-m7gk4\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") " pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:02.993037 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.993025    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-tls-certs\") pod \"auth-test-model-04464b-kserve-6ddb8bb859-m7gk4\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") " pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:02.993197 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.993059    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-tmp-dir\") pod \"auth-test-model-04464b-kserve-6ddb8bb859-m7gk4\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") " pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:02.993197 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.993181    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-model-cache\") pod \"auth-test-model-04464b-kserve-6ddb8bb859-m7gk4\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") " pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:02.996632 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.996614    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b21d1ff2-7ccvz"
Sep 15 07:46:02.996744 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.996663    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b21d1ff2-7ccvz" event={"ID":"c26856b1-41e0-408b-96f3-3148ebdfd757","Type":"ContainerDied","Data":"734fda5853915a5f93503c4001305406cc97c9287a41ddac861e5b867ddaf7f5"}
Sep 15 07:46:02.996744 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:02.996705    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="734fda5853915a5f93503c4001305406cc97c9287a41ddac861e5b867ddaf7f5"
Sep 15 07:46:03.094097 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:03.094062    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-model-cache\") pod \"auth-test-model-04464b-kserve-6ddb8bb859-m7gk4\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") " pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:03.094097 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:03.094101    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksrht\" (UniqueName: \"kubernetes.io/projected/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-kube-api-access-ksrht\") pod \"auth-test-model-04464b-kserve-6ddb8bb859-m7gk4\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") " pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:03.094389 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:03.094138    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-home\") pod \"auth-test-model-04464b-kserve-6ddb8bb859-m7gk4\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") " pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:03.094389 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:03.094182    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-dshm\") pod \"auth-test-model-04464b-kserve-6ddb8bb859-m7gk4\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") " pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:03.094389 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:03.094208    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-tls-certs\") pod \"auth-test-model-04464b-kserve-6ddb8bb859-m7gk4\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") " pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:03.094548 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:03.094404    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-tmp-dir\") pod \"auth-test-model-04464b-kserve-6ddb8bb859-m7gk4\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") " pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:03.094612 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:03.094540    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-model-cache\") pod \"auth-test-model-04464b-kserve-6ddb8bb859-m7gk4\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") " pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:03.094612 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:03.094572    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-home\") pod \"auth-test-model-04464b-kserve-6ddb8bb859-m7gk4\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") " pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:03.094735 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:03.094714    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-tmp-dir\") pod \"auth-test-model-04464b-kserve-6ddb8bb859-m7gk4\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") " pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:03.096605 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:03.096580    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-dshm\") pod \"auth-test-model-04464b-kserve-6ddb8bb859-m7gk4\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") " pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:03.096755 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:03.096738    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-tls-certs\") pod \"auth-test-model-04464b-kserve-6ddb8bb859-m7gk4\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") " pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:03.103780 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:03.103756    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksrht\" (UniqueName: \"kubernetes.io/projected/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-kube-api-access-ksrht\") pod \"auth-test-model-04464b-kserve-6ddb8bb859-m7gk4\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") " pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:03.215508 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:03.215440    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:03.489148 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:03.489119    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"]
Sep 15 07:46:03.491087 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:46:03.491064    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea796e0e_601f_4aa1_aa8e_e01673fbe40f.slice/crio-f6810cc6229a4f1dd48c3252c8b725f501285683e0b069a21f974b70ba4a8e3d WatchSource:0}: Error finding container f6810cc6229a4f1dd48c3252c8b725f501285683e0b069a21f974b70ba4a8e3d: Status 404 returned error can't find the container with id f6810cc6229a4f1dd48c3252c8b725f501285683e0b069a21f974b70ba4a8e3d
Sep 15 07:46:04.003637 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:04.003595    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4" event={"ID":"ea796e0e-601f-4aa1-aa8e-e01673fbe40f","Type":"ContainerStarted","Data":"386dbd0e5945cf6c7e0c8944e1803aa302b0ceb2c601ee2ef7b7d3a25a780880"}
Sep 15 07:46:04.003637 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:04.003638    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4" event={"ID":"ea796e0e-601f-4aa1-aa8e-e01673fbe40f","Type":"ContainerStarted","Data":"f6810cc6229a4f1dd48c3252c8b725f501285683e0b069a21f974b70ba4a8e3d"}
Sep 15 07:46:04.021743 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:04.021693    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4" podStartSLOduration=2.021676971 podStartE2EDuration="2.021676971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:46:04.01897506 +0000 UTC m=+1668.565545160" watchObservedRunningTime="2026-09-15 07:46:04.021676971 +0000 UTC m=+1668.568247069"
Sep 15 07:46:06.896876 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:06.896838    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"]
Sep 15 07:46:13.216243 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:13.216201    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:13.216861 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:13.216290    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:13.229047 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:13.229021    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:14.076044 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:14.076013    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:46:15.050940 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:15.050893    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b21d1ff2-7ccvz"]
Sep 15 07:46:15.053460 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:15.053442    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b21d1ff2-7ccvz"]
Sep 15 07:46:15.909839 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:15.909802    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c26856b1-41e0-408b-96f3-3148ebdfd757" path="/var/lib/kubelet/pods/c26856b1-41e0-408b-96f3-3148ebdfd757/volumes"
Sep 15 07:46:21.917766 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:21.917709    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg" podUID="f6be841e-c24f-4b66-ad25-4528713aef88" containerName="main" containerID="cri-o://ef94bb7760fc3bc9868071df0a743deb7c2c736e811248c73bf94f11a4e547d0" gracePeriod=45
Sep 15 07:46:22.115091 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:22.115052    2571 generic.go:356] "Generic (PLEG): container finished" podID="f6be841e-c24f-4b66-ad25-4528713aef88" containerID="ef94bb7760fc3bc9868071df0a743deb7c2c736e811248c73bf94f11a4e547d0" exitCode=0
Sep 15 07:46:22.115249 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:22.115123    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg" event={"ID":"f6be841e-c24f-4b66-ad25-4528713aef88","Type":"ContainerDied","Data":"ef94bb7760fc3bc9868071df0a743deb7c2c736e811248c73bf94f11a4e547d0"}
Sep 15 07:46:22.168623 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:22.168562    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:46:22.290402 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:22.290356    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-home\") pod \"f6be841e-c24f-4b66-ad25-4528713aef88\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") "
Sep 15 07:46:22.290595 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:22.290445    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6be841e-c24f-4b66-ad25-4528713aef88-tls-certs\") pod \"f6be841e-c24f-4b66-ad25-4528713aef88\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") "
Sep 15 07:46:22.290595 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:22.290476    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-model-cache\") pod \"f6be841e-c24f-4b66-ad25-4528713aef88\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") "
Sep 15 07:46:22.290595 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:22.290518    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbx4s\" (UniqueName: \"kubernetes.io/projected/f6be841e-c24f-4b66-ad25-4528713aef88-kube-api-access-mbx4s\") pod \"f6be841e-c24f-4b66-ad25-4528713aef88\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") "
Sep 15 07:46:22.290778 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:22.290651    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-dshm\") pod \"f6be841e-c24f-4b66-ad25-4528713aef88\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") "
Sep 15 07:46:22.290778 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:22.290765    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-tmp-dir\") pod \"f6be841e-c24f-4b66-ad25-4528713aef88\" (UID: \"f6be841e-c24f-4b66-ad25-4528713aef88\") "
Sep 15 07:46:22.290881 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:22.290650    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-home" (OuterVolumeSpecName: "home") pod "f6be841e-c24f-4b66-ad25-4528713aef88" (UID: "f6be841e-c24f-4b66-ad25-4528713aef88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:46:22.290881 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:22.290785    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6be841e-c24f-4b66-ad25-4528713aef88" (UID: "f6be841e-c24f-4b66-ad25-4528713aef88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:46:22.291031 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:22.291008    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6be841e-c24f-4b66-ad25-4528713aef88" (UID: "f6be841e-c24f-4b66-ad25-4528713aef88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:46:22.291197 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:22.291156    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-tmp-dir\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:46:22.291197 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:22.291179    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-home\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:46:22.291197 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:22.291193    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-model-cache\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:46:22.292753 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:22.292728    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6be841e-c24f-4b66-ad25-4528713aef88-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6be841e-c24f-4b66-ad25-4528713aef88" (UID: "f6be841e-c24f-4b66-ad25-4528713aef88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:46:22.292753 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:22.292735    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-dshm" (OuterVolumeSpecName: "dshm") pod "f6be841e-c24f-4b66-ad25-4528713aef88" (UID: "f6be841e-c24f-4b66-ad25-4528713aef88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:46:22.292881 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:22.292755    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6be841e-c24f-4b66-ad25-4528713aef88-kube-api-access-mbx4s" (OuterVolumeSpecName: "kube-api-access-mbx4s") pod "f6be841e-c24f-4b66-ad25-4528713aef88" (UID: "f6be841e-c24f-4b66-ad25-4528713aef88"). InnerVolumeSpecName "kube-api-access-mbx4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:46:22.392018 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:22.391961    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6be841e-c24f-4b66-ad25-4528713aef88-tls-certs\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:46:22.392018 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:22.392021    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbx4s\" (UniqueName: \"kubernetes.io/projected/f6be841e-c24f-4b66-ad25-4528713aef88-kube-api-access-mbx4s\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:46:22.392232 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:22.392038    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6be841e-c24f-4b66-ad25-4528713aef88-dshm\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:46:23.123304 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:23.123273    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"
Sep 15 07:46:23.123304 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:23.123297    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg" event={"ID":"f6be841e-c24f-4b66-ad25-4528713aef88","Type":"ContainerDied","Data":"26b9641de28aeda2af2c62142c9104c850d820a6979a6f89a8321bc4d6b2f8a7"}
Sep 15 07:46:23.123783 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:23.123346    2571 scope.go:117] "RemoveContainer" containerID="ef94bb7760fc3bc9868071df0a743deb7c2c736e811248c73bf94f11a4e547d0"
Sep 15 07:46:23.146341 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:23.146316    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"]
Sep 15 07:46:23.151696 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:23.151674    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/e2e-conflict-b-30d020fe-kserve-56767dbb67-vsdkg"]
Sep 15 07:46:23.910850 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:23.910814    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6be841e-c24f-4b66-ad25-4528713aef88" path="/var/lib/kubelet/pods/f6be841e-c24f-4b66-ad25-4528713aef88/volumes"
Sep 15 07:46:31.737903 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:31.737861    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"]
Sep 15 07:46:46.758902 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:46.758838    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk" podUID="16abe562-4784-49d5-9b9a-c46ffc4900d2" containerName="main" containerID="cri-o://b7e65dae342474c7ffbd34c0a0b6ee67b445a838efed2a81d449353de90d116b" gracePeriod=45
Sep 15 07:46:47.019405 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.019335    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:46:47.135047 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.135013    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqmk2\" (UniqueName: \"kubernetes.io/projected/16abe562-4784-49d5-9b9a-c46ffc4900d2-kube-api-access-kqmk2\") pod \"16abe562-4784-49d5-9b9a-c46ffc4900d2\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") "
Sep 15 07:46:47.135047 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.135053    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-model-cache\") pod \"16abe562-4784-49d5-9b9a-c46ffc4900d2\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") "
Sep 15 07:46:47.135341 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.135097    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-tmp-dir\") pod \"16abe562-4784-49d5-9b9a-c46ffc4900d2\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") "
Sep 15 07:46:47.135341 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.135146    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-dshm\") pod \"16abe562-4784-49d5-9b9a-c46ffc4900d2\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") "
Sep 15 07:46:47.135341 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.135235    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16abe562-4784-49d5-9b9a-c46ffc4900d2-tls-certs\") pod \"16abe562-4784-49d5-9b9a-c46ffc4900d2\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") "
Sep 15 07:46:47.135489 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.135350    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-home\") pod \"16abe562-4784-49d5-9b9a-c46ffc4900d2\" (UID: \"16abe562-4784-49d5-9b9a-c46ffc4900d2\") "
Sep 15 07:46:47.135489 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.135372    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "16abe562-4784-49d5-9b9a-c46ffc4900d2" (UID: "16abe562-4784-49d5-9b9a-c46ffc4900d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:46:47.135489 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.135418    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "16abe562-4784-49d5-9b9a-c46ffc4900d2" (UID: "16abe562-4784-49d5-9b9a-c46ffc4900d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:46:47.135660 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.135633    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-home" (OuterVolumeSpecName: "home") pod "16abe562-4784-49d5-9b9a-c46ffc4900d2" (UID: "16abe562-4784-49d5-9b9a-c46ffc4900d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:46:47.135715 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.135659    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-model-cache\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:46:47.135715 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.135677    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-tmp-dir\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:46:47.137289 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.137244    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16abe562-4784-49d5-9b9a-c46ffc4900d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16abe562-4784-49d5-9b9a-c46ffc4900d2" (UID: "16abe562-4784-49d5-9b9a-c46ffc4900d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:46:47.137376 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.137285    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16abe562-4784-49d5-9b9a-c46ffc4900d2-kube-api-access-kqmk2" (OuterVolumeSpecName: "kube-api-access-kqmk2") pod "16abe562-4784-49d5-9b9a-c46ffc4900d2" (UID: "16abe562-4784-49d5-9b9a-c46ffc4900d2"). InnerVolumeSpecName "kube-api-access-kqmk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:46:47.137376 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.137362    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-dshm" (OuterVolumeSpecName: "dshm") pod "16abe562-4784-49d5-9b9a-c46ffc4900d2" (UID: "16abe562-4784-49d5-9b9a-c46ffc4900d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:46:47.237221 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.237181    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-home\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:46:47.237221 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.237213    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqmk2\" (UniqueName: \"kubernetes.io/projected/16abe562-4784-49d5-9b9a-c46ffc4900d2-kube-api-access-kqmk2\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:46:47.237221 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.237223    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16abe562-4784-49d5-9b9a-c46ffc4900d2-dshm\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:46:47.237221 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.237233    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16abe562-4784-49d5-9b9a-c46ffc4900d2-tls-certs\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:46:47.267330 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.267294    2571 generic.go:356] "Generic (PLEG): container finished" podID="16abe562-4784-49d5-9b9a-c46ffc4900d2" containerID="b7e65dae342474c7ffbd34c0a0b6ee67b445a838efed2a81d449353de90d116b" exitCode=0
Sep 15 07:46:47.267507 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.267335    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"
Sep 15 07:46:47.267507 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.267375    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk" event={"ID":"16abe562-4784-49d5-9b9a-c46ffc4900d2","Type":"ContainerDied","Data":"b7e65dae342474c7ffbd34c0a0b6ee67b445a838efed2a81d449353de90d116b"}
Sep 15 07:46:47.267507 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.267411    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk" event={"ID":"16abe562-4784-49d5-9b9a-c46ffc4900d2","Type":"ContainerDied","Data":"da6c9243aee8e2f67d68482a6582b0f6b1ec7c2102a207e73046bccc74e1a6f9"}
Sep 15 07:46:47.267507 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.267428    2571 scope.go:117] "RemoveContainer" containerID="b7e65dae342474c7ffbd34c0a0b6ee67b445a838efed2a81d449353de90d116b"
Sep 15 07:46:47.278035 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.278018    2571 scope.go:117] "RemoveContainer" containerID="b7e65dae342474c7ffbd34c0a0b6ee67b445a838efed2a81d449353de90d116b"
Sep 15 07:46:47.278321 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:46:47.278299    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7e65dae342474c7ffbd34c0a0b6ee67b445a838efed2a81d449353de90d116b\": container with ID starting with b7e65dae342474c7ffbd34c0a0b6ee67b445a838efed2a81d449353de90d116b not found: ID does not exist" containerID="b7e65dae342474c7ffbd34c0a0b6ee67b445a838efed2a81d449353de90d116b"
Sep 15 07:46:47.278404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.278328    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e65dae342474c7ffbd34c0a0b6ee67b445a838efed2a81d449353de90d116b"} err="failed to get container status \"b7e65dae342474c7ffbd34c0a0b6ee67b445a838efed2a81d449353de90d116b\": rpc error: code = NotFound desc = could not find container \"b7e65dae342474c7ffbd34c0a0b6ee67b445a838efed2a81d449353de90d116b\": container with ID starting with b7e65dae342474c7ffbd34c0a0b6ee67b445a838efed2a81d449353de90d116b not found: ID does not exist"
Sep 15 07:46:47.289173 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.289132    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"]
Sep 15 07:46:47.294509 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.294483    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/e2e-conflict-a-30d020fe-kserve-64855995df-ssklk"]
Sep 15 07:46:47.910135 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:47.910101    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16abe562-4784-49d5-9b9a-c46ffc4900d2" path="/var/lib/kubelet/pods/16abe562-4784-49d5-9b9a-c46ffc4900d2/volumes"
Sep 15 07:46:51.210731 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.210688    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"]
Sep 15 07:46:51.211367 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.211346    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6be841e-c24f-4b66-ad25-4528713aef88" containerName="main"
Sep 15 07:46:51.211367 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.211369    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6be841e-c24f-4b66-ad25-4528713aef88" containerName="main"
Sep 15 07:46:51.211535 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.211419    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16abe562-4784-49d5-9b9a-c46ffc4900d2" containerName="main"
Sep 15 07:46:51.211535 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.211428    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="16abe562-4784-49d5-9b9a-c46ffc4900d2" containerName="main"
Sep 15 07:46:51.211631 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.211547    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="16abe562-4784-49d5-9b9a-c46ffc4900d2" containerName="main"
Sep 15 07:46:51.211631 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.211561    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6be841e-c24f-4b66-ad25-4528713aef88" containerName="main"
Sep 15 07:46:51.214509 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.214488    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:46:51.217404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.217381    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f56165\"/\"distinct-w4-f56165-kserve-self-signed-certs\""
Sep 15 07:46:51.222816 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.222796    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"]
Sep 15 07:46:51.281573 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.281537    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-tls-certs\") pod \"distinct-w4-f56165-kserve-7d6bbd46db-fr4rk\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") " pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:46:51.281722 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.281651    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-home\") pod \"distinct-w4-f56165-kserve-7d6bbd46db-fr4rk\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") " pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:46:51.281722 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.281683    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvc6t\" (UniqueName: \"kubernetes.io/projected/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-kube-api-access-hvc6t\") pod \"distinct-w4-f56165-kserve-7d6bbd46db-fr4rk\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") " pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:46:51.281864 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.281729    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-tmp-dir\") pod \"distinct-w4-f56165-kserve-7d6bbd46db-fr4rk\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") " pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:46:51.281926 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.281866    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-dshm\") pod \"distinct-w4-f56165-kserve-7d6bbd46db-fr4rk\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") " pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:46:51.281926 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.281908    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-model-cache\") pod \"distinct-w4-f56165-kserve-7d6bbd46db-fr4rk\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") " pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:46:51.383597 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.383555    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-dshm\") pod \"distinct-w4-f56165-kserve-7d6bbd46db-fr4rk\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") " pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:46:51.383597 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.383598    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-model-cache\") pod \"distinct-w4-f56165-kserve-7d6bbd46db-fr4rk\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") " pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:46:51.383881 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.383687    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-tls-certs\") pod \"distinct-w4-f56165-kserve-7d6bbd46db-fr4rk\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") " pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:46:51.383881 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.383747    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-home\") pod \"distinct-w4-f56165-kserve-7d6bbd46db-fr4rk\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") " pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:46:51.383881 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.383775    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvc6t\" (UniqueName: \"kubernetes.io/projected/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-kube-api-access-hvc6t\") pod \"distinct-w4-f56165-kserve-7d6bbd46db-fr4rk\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") " pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:46:51.383881 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.383827    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-tmp-dir\") pod \"distinct-w4-f56165-kserve-7d6bbd46db-fr4rk\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") " pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:46:51.384107 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.384082    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-model-cache\") pod \"distinct-w4-f56165-kserve-7d6bbd46db-fr4rk\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") " pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:46:51.384182 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.384101    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-home\") pod \"distinct-w4-f56165-kserve-7d6bbd46db-fr4rk\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") " pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:46:51.384182 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.384176    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-tmp-dir\") pod \"distinct-w4-f56165-kserve-7d6bbd46db-fr4rk\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") " pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:46:51.385949 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.385917    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-dshm\") pod \"distinct-w4-f56165-kserve-7d6bbd46db-fr4rk\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") " pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:46:51.386179 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.386157    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-tls-certs\") pod \"distinct-w4-f56165-kserve-7d6bbd46db-fr4rk\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") " pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:46:51.392024 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.392001    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvc6t\" (UniqueName: \"kubernetes.io/projected/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-kube-api-access-hvc6t\") pod \"distinct-w4-f56165-kserve-7d6bbd46db-fr4rk\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") " pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:46:51.524182 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.524095    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:46:51.797325 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:51.797292    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"]
Sep 15 07:46:51.798736 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:46:51.798711    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod411641e0_2bbf_4ddf_bbf4_fce8b8ed41f0.slice/crio-ef6a665c395cb9e9a0dc3ad0a9bb7f48894b21f0af42b420fa28b8c1c806e698 WatchSource:0}: Error finding container ef6a665c395cb9e9a0dc3ad0a9bb7f48894b21f0af42b420fa28b8c1c806e698: Status 404 returned error can't find the container with id ef6a665c395cb9e9a0dc3ad0a9bb7f48894b21f0af42b420fa28b8c1c806e698
Sep 15 07:46:52.302312 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:52.302275    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk" event={"ID":"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0","Type":"ContainerStarted","Data":"1d5485b158658939d03973841ba00a165ab53751d0be1dc8d36ed9720e5988c8"}
Sep 15 07:46:52.302690 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:52.302321    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk" event={"ID":"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0","Type":"ContainerStarted","Data":"ef6a665c395cb9e9a0dc3ad0a9bb7f48894b21f0af42b420fa28b8c1c806e698"}
Sep 15 07:46:52.322568 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:46:52.322515    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk" podStartSLOduration=1.3224996820000001 podStartE2EDuration="1.322499682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:46:52.31946563 +0000 UTC m=+1716.866035749" watchObservedRunningTime="2026-09-15 07:46:52.322499682 +0000 UTC m=+1716.869069781"
Sep 15 07:47:01.524622 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:01.524577    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:47:01.525049 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:01.524641    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:47:01.537725 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:01.537701    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:47:02.371985 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:02.371951    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:47:04.723296 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.723242    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"]
Sep 15 07:47:04.727236 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.727212    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:04.729697 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.729674    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e3a482\"/\"kube-root-ca.crt\""
Sep 15 07:47:04.730534 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.730512    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e3a482\"/\"auth-test-model-e3a482-kserve-self-signed-certs\""
Sep 15 07:47:04.730628 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.730533    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e3a482\"/\"openshift-service-ca.crt\""
Sep 15 07:47:04.730628 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.730592    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e3a482\"/\"default-dockercfg-jgj2b\""
Sep 15 07:47:04.735117 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.735097    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"]
Sep 15 07:47:04.823030 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.822985    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-model-cache\") pod \"auth-test-model-e3a482-kserve-769cb764f9-qrhjb\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:04.823030 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.823049    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5d46da-426d-4dca-bc70-982b481cb5d8-tls-certs\") pod \"auth-test-model-e3a482-kserve-769cb764f9-qrhjb\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:04.823345 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.823072    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv79n\" (UniqueName: \"kubernetes.io/projected/bf5d46da-426d-4dca-bc70-982b481cb5d8-kube-api-access-mv79n\") pod \"auth-test-model-e3a482-kserve-769cb764f9-qrhjb\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:04.823345 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.823115    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-dshm\") pod \"auth-test-model-e3a482-kserve-769cb764f9-qrhjb\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:04.823345 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.823142    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-home\") pod \"auth-test-model-e3a482-kserve-769cb764f9-qrhjb\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:04.823345 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.823243    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-tmp-dir\") pod \"auth-test-model-e3a482-kserve-769cb764f9-qrhjb\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:04.924156 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.924123    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-model-cache\") pod \"auth-test-model-e3a482-kserve-769cb764f9-qrhjb\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:04.924400 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.924166    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5d46da-426d-4dca-bc70-982b481cb5d8-tls-certs\") pod \"auth-test-model-e3a482-kserve-769cb764f9-qrhjb\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:04.924400 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.924210    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv79n\" (UniqueName: \"kubernetes.io/projected/bf5d46da-426d-4dca-bc70-982b481cb5d8-kube-api-access-mv79n\") pod \"auth-test-model-e3a482-kserve-769cb764f9-qrhjb\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:04.924400 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.924314    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-dshm\") pod \"auth-test-model-e3a482-kserve-769cb764f9-qrhjb\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:04.924400 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.924349    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-home\") pod \"auth-test-model-e3a482-kserve-769cb764f9-qrhjb\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:04.924400 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.924401    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-tmp-dir\") pod \"auth-test-model-e3a482-kserve-769cb764f9-qrhjb\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:04.924795 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.924768    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-home\") pod \"auth-test-model-e3a482-kserve-769cb764f9-qrhjb\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:04.924904 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.924815    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-model-cache\") pod \"auth-test-model-e3a482-kserve-769cb764f9-qrhjb\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:04.924904 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.924866    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-tmp-dir\") pod \"auth-test-model-e3a482-kserve-769cb764f9-qrhjb\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:04.927165 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.927143    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-dshm\") pod \"auth-test-model-e3a482-kserve-769cb764f9-qrhjb\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:04.927392 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.927378    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5d46da-426d-4dca-bc70-982b481cb5d8-tls-certs\") pod \"auth-test-model-e3a482-kserve-769cb764f9-qrhjb\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:04.933017 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:04.932994    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv79n\" (UniqueName: \"kubernetes.io/projected/bf5d46da-426d-4dca-bc70-982b481cb5d8-kube-api-access-mv79n\") pod \"auth-test-model-e3a482-kserve-769cb764f9-qrhjb\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:05.039359 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:05.039226    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:05.312932 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:05.312902    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"]
Sep 15 07:47:05.314151 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:47:05.314125    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf5d46da_426d_4dca_bc70_982b481cb5d8.slice/crio-80ad3b02acdf0e5f3d42864489619632b5119c8050b83aa3589d6e20f7399a79 WatchSource:0}: Error finding container 80ad3b02acdf0e5f3d42864489619632b5119c8050b83aa3589d6e20f7399a79: Status 404 returned error can't find the container with id 80ad3b02acdf0e5f3d42864489619632b5119c8050b83aa3589d6e20f7399a79
Sep 15 07:47:05.381074 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:05.381048    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb" event={"ID":"bf5d46da-426d-4dca-bc70-982b481cb5d8","Type":"ContainerStarted","Data":"80ad3b02acdf0e5f3d42864489619632b5119c8050b83aa3589d6e20f7399a79"}
Sep 15 07:47:05.396850 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:05.396801    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb" podStartSLOduration=1.39678528 podStartE2EDuration="1.39678528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:47:05.395874096 +0000 UTC m=+1729.942444216" watchObservedRunningTime="2026-09-15 07:47:05.39678528 +0000 UTC m=+1729.943355376"
Sep 15 07:47:06.388816 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:06.388776    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb" event={"ID":"bf5d46da-426d-4dca-bc70-982b481cb5d8","Type":"ContainerStarted","Data":"e0ffa50405e1ec3a91b4c6dd38b7d4e5f415d736661f67db223882b363b3c634"}
Sep 15 07:47:15.039751 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:15.039714    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:15.040118 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:15.039763    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:15.052843 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:15.052814    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:15.458615 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:15.458584    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:47:16.075442 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:47:16.075410    2571 scope.go:117] "RemoveContainer" containerID="faba866db02fefe5f003c108590b572aed495491b9bb656a4068defe05cb426c"
Sep 15 07:48:14.484973 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:14.484932    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"]
Sep 15 07:48:14.543510 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:14.543406    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"]
Sep 15 07:48:25.694615 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:25.694578    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2735c5-29824305-wfzgm"]
Sep 15 07:48:25.699807 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:25.699778    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2735c5-29824305-wfzgm"]
Sep 15 07:48:25.735548 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:25.735513    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww"]
Sep 15 07:48:25.735856 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:25.735828    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww" podUID="d6c1f080-389f-4ab8-abd7-e62804e7302f" containerName="payload-pre-processing" containerID="cri-o://f74c951805a2495a829abe383ce6a186771e10e4d759c365d9511c92df071e0b" gracePeriod=30
Sep 15 07:48:25.875646 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:25.875606    2571 generic.go:356] "Generic (PLEG): container finished" podID="d6c1f080-389f-4ab8-abd7-e62804e7302f" containerID="f74c951805a2495a829abe383ce6a186771e10e4d759c365d9511c92df071e0b" exitCode=0
Sep 15 07:48:25.875839 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:25.875677    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww" event={"ID":"d6c1f080-389f-4ab8-abd7-e62804e7302f","Type":"ContainerDied","Data":"f74c951805a2495a829abe383ce6a186771e10e4d759c365d9511c92df071e0b"}
Sep 15 07:48:25.913179 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:25.913150    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ace9f8ee-1513-483d-b257-5c3f830c66b4" path="/var/lib/kubelet/pods/ace9f8ee-1513-483d-b257-5c3f830c66b4/volumes"
Sep 15 07:48:26.002478 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:26.002450    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww"
Sep 15 07:48:26.049144 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:26.049094    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5mfv\" (UniqueName: \"kubernetes.io/projected/d6c1f080-389f-4ab8-abd7-e62804e7302f-kube-api-access-k5mfv\") pod \"d6c1f080-389f-4ab8-abd7-e62804e7302f\" (UID: \"d6c1f080-389f-4ab8-abd7-e62804e7302f\") "
Sep 15 07:48:26.049365 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:26.049249    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c1f080-389f-4ab8-abd7-e62804e7302f-plugins-config-volume\") pod \"d6c1f080-389f-4ab8-abd7-e62804e7302f\" (UID: \"d6c1f080-389f-4ab8-abd7-e62804e7302f\") "
Sep 15 07:48:26.049746 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:26.049721    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c1f080-389f-4ab8-abd7-e62804e7302f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d6c1f080-389f-4ab8-abd7-e62804e7302f" (UID: "d6c1f080-389f-4ab8-abd7-e62804e7302f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:48:26.051305 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:26.051273    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c1f080-389f-4ab8-abd7-e62804e7302f-kube-api-access-k5mfv" (OuterVolumeSpecName: "kube-api-access-k5mfv") pod "d6c1f080-389f-4ab8-abd7-e62804e7302f" (UID: "d6c1f080-389f-4ab8-abd7-e62804e7302f"). InnerVolumeSpecName "kube-api-access-k5mfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:48:26.150632 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:26.150597    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c1f080-389f-4ab8-abd7-e62804e7302f-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:48:26.150632 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:26.150626    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5mfv\" (UniqueName: \"kubernetes.io/projected/d6c1f080-389f-4ab8-abd7-e62804e7302f-kube-api-access-k5mfv\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:48:26.883424 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:26.883385    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww" event={"ID":"d6c1f080-389f-4ab8-abd7-e62804e7302f","Type":"ContainerDied","Data":"69475d03b6e9a85f71b952312d5d88e887fc3b9321b5929b79b0c83b3d26a9f4"}
Sep 15 07:48:26.883424 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:26.883408    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww"
Sep 15 07:48:26.883896 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:26.883451    2571 scope.go:117] "RemoveContainer" containerID="f74c951805a2495a829abe383ce6a186771e10e4d759c365d9511c92df071e0b"
Sep 15 07:48:26.907109 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:26.907082    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww"]
Sep 15 07:48:26.913576 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:26.913552    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2735c5-5c8d5d4bd6-nmdww"]
Sep 15 07:48:27.909880 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:27.909830    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6c1f080-389f-4ab8-abd7-e62804e7302f" path="/var/lib/kubelet/pods/d6c1f080-389f-4ab8-abd7-e62804e7302f/volumes"
Sep 15 07:48:29.504656 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.504611    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp" podUID="6c4190df-9881-42fc-bda5-b73881600bf6" containerName="main" containerID="cri-o://cf3f6c8128999ad212b987fd45e888de8f8d61e161e971ebdef341af564d40f6" gracePeriod=45
Sep 15 07:48:29.562815 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.562777    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7" podUID="dff30f35-72cb-4fe8-b87d-8bc20401fd8d" containerName="main" containerID="cri-o://30fa8ce2066dad24915e2a757a4b60230c2d3bcea0902e591e4600e8e0a9690b" gracePeriod=45
Sep 15 07:48:29.849320 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.849290    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:48:29.852398 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.852366    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:48:29.890591 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.890557    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-dshm\") pod \"6c4190df-9881-42fc-bda5-b73881600bf6\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") "
Sep 15 07:48:29.890767 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.890609    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-home\") pod \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") "
Sep 15 07:48:29.890767 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.890639    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gkg8\" (UniqueName: \"kubernetes.io/projected/6c4190df-9881-42fc-bda5-b73881600bf6-kube-api-access-7gkg8\") pod \"6c4190df-9881-42fc-bda5-b73881600bf6\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") "
Sep 15 07:48:29.890767 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.890700    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-tmp-dir\") pod \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") "
Sep 15 07:48:29.892375 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.891243    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57z9l\" (UniqueName: \"kubernetes.io/projected/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-kube-api-access-57z9l\") pod \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") "
Sep 15 07:48:29.892375 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.891333    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-tls-certs\") pod \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") "
Sep 15 07:48:29.892375 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.891393    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-tmp-dir\") pod \"6c4190df-9881-42fc-bda5-b73881600bf6\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") "
Sep 15 07:48:29.892375 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.891478    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-model-cache\") pod \"6c4190df-9881-42fc-bda5-b73881600bf6\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") "
Sep 15 07:48:29.892375 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.891510    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-dshm\") pod \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") "
Sep 15 07:48:29.892375 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.891538    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-home\") pod \"6c4190df-9881-42fc-bda5-b73881600bf6\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") "
Sep 15 07:48:29.892375 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.891573    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-model-cache\") pod \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\" (UID: \"dff30f35-72cb-4fe8-b87d-8bc20401fd8d\") "
Sep 15 07:48:29.892375 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.891602    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4190df-9881-42fc-bda5-b73881600bf6-tls-certs\") pod \"6c4190df-9881-42fc-bda5-b73881600bf6\" (UID: \"6c4190df-9881-42fc-bda5-b73881600bf6\") "
Sep 15 07:48:29.892375 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.891697    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-home" (OuterVolumeSpecName: "home") pod "dff30f35-72cb-4fe8-b87d-8bc20401fd8d" (UID: "dff30f35-72cb-4fe8-b87d-8bc20401fd8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:48:29.892375 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.891718    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dff30f35-72cb-4fe8-b87d-8bc20401fd8d" (UID: "dff30f35-72cb-4fe8-b87d-8bc20401fd8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:48:29.892375 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.891968    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-home" (OuterVolumeSpecName: "home") pod "6c4190df-9881-42fc-bda5-b73881600bf6" (UID: "6c4190df-9881-42fc-bda5-b73881600bf6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:48:29.892375 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.891990    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c4190df-9881-42fc-bda5-b73881600bf6" (UID: "6c4190df-9881-42fc-bda5-b73881600bf6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:48:29.892375 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.892074    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-home\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:48:29.892375 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.892093    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-tmp-dir\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:48:29.892375 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.892105    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-tmp-dir\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:48:29.892375 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.892147    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-home\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:48:29.892375 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.892184    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-model-cache" (OuterVolumeSpecName: "model-cache") pod "dff30f35-72cb-4fe8-b87d-8bc20401fd8d" (UID: "dff30f35-72cb-4fe8-b87d-8bc20401fd8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:48:29.892375 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.892343    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c4190df-9881-42fc-bda5-b73881600bf6" (UID: "6c4190df-9881-42fc-bda5-b73881600bf6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:48:29.894498 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.894472    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-dshm" (OuterVolumeSpecName: "dshm") pod "dff30f35-72cb-4fe8-b87d-8bc20401fd8d" (UID: "dff30f35-72cb-4fe8-b87d-8bc20401fd8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:48:29.894586 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.894492    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c4190df-9881-42fc-bda5-b73881600bf6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c4190df-9881-42fc-bda5-b73881600bf6" (UID: "6c4190df-9881-42fc-bda5-b73881600bf6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:48:29.894654 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.894597    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-dshm" (OuterVolumeSpecName: "dshm") pod "6c4190df-9881-42fc-bda5-b73881600bf6" (UID: "6c4190df-9881-42fc-bda5-b73881600bf6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:48:29.895063 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.895036    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dff30f35-72cb-4fe8-b87d-8bc20401fd8d" (UID: "dff30f35-72cb-4fe8-b87d-8bc20401fd8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:48:29.895063 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.895052    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c4190df-9881-42fc-bda5-b73881600bf6-kube-api-access-7gkg8" (OuterVolumeSpecName: "kube-api-access-7gkg8") pod "6c4190df-9881-42fc-bda5-b73881600bf6" (UID: "6c4190df-9881-42fc-bda5-b73881600bf6"). InnerVolumeSpecName "kube-api-access-7gkg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:48:29.895908 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.895883    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-kube-api-access-57z9l" (OuterVolumeSpecName: "kube-api-access-57z9l") pod "dff30f35-72cb-4fe8-b87d-8bc20401fd8d" (UID: "dff30f35-72cb-4fe8-b87d-8bc20401fd8d"). InnerVolumeSpecName "kube-api-access-57z9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:48:29.922052 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.921815    2571 generic.go:356] "Generic (PLEG): container finished" podID="dff30f35-72cb-4fe8-b87d-8bc20401fd8d" containerID="30fa8ce2066dad24915e2a757a4b60230c2d3bcea0902e591e4600e8e0a9690b" exitCode=0
Sep 15 07:48:29.922052 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.921873    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7" event={"ID":"dff30f35-72cb-4fe8-b87d-8bc20401fd8d","Type":"ContainerDied","Data":"30fa8ce2066dad24915e2a757a4b60230c2d3bcea0902e591e4600e8e0a9690b"}
Sep 15 07:48:29.922052 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.922001    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7" event={"ID":"dff30f35-72cb-4fe8-b87d-8bc20401fd8d","Type":"ContainerDied","Data":"2c5c660735ae1c8b33bc2cd6553cefe7af450af39d34c3799cea5581091e416a"}
Sep 15 07:48:29.922052 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.922027    2571 scope.go:117] "RemoveContainer" containerID="30fa8ce2066dad24915e2a757a4b60230c2d3bcea0902e591e4600e8e0a9690b"
Sep 15 07:48:29.922411 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.921881    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"
Sep 15 07:48:29.927634 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.924739    2571 generic.go:356] "Generic (PLEG): container finished" podID="6c4190df-9881-42fc-bda5-b73881600bf6" containerID="cf3f6c8128999ad212b987fd45e888de8f8d61e161e971ebdef341af564d40f6" exitCode=0
Sep 15 07:48:29.927634 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.924789    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp" event={"ID":"6c4190df-9881-42fc-bda5-b73881600bf6","Type":"ContainerDied","Data":"cf3f6c8128999ad212b987fd45e888de8f8d61e161e971ebdef341af564d40f6"}
Sep 15 07:48:29.927634 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.924814    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp" event={"ID":"6c4190df-9881-42fc-bda5-b73881600bf6","Type":"ContainerDied","Data":"1f88bba3d23f0307ea08c1f2180c896f8a47da9b2d923e46d69297d17cd4f6b2"}
Sep 15 07:48:29.927634 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.924880    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"
Sep 15 07:48:29.938191 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.937994    2571 scope.go:117] "RemoveContainer" containerID="30fa8ce2066dad24915e2a757a4b60230c2d3bcea0902e591e4600e8e0a9690b"
Sep 15 07:48:29.939872 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:48:29.939847    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30fa8ce2066dad24915e2a757a4b60230c2d3bcea0902e591e4600e8e0a9690b\": container with ID starting with 30fa8ce2066dad24915e2a757a4b60230c2d3bcea0902e591e4600e8e0a9690b not found: ID does not exist" containerID="30fa8ce2066dad24915e2a757a4b60230c2d3bcea0902e591e4600e8e0a9690b"
Sep 15 07:48:29.939964 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.939884    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30fa8ce2066dad24915e2a757a4b60230c2d3bcea0902e591e4600e8e0a9690b"} err="failed to get container status \"30fa8ce2066dad24915e2a757a4b60230c2d3bcea0902e591e4600e8e0a9690b\": rpc error: code = NotFound desc = could not find container \"30fa8ce2066dad24915e2a757a4b60230c2d3bcea0902e591e4600e8e0a9690b\": container with ID starting with 30fa8ce2066dad24915e2a757a4b60230c2d3bcea0902e591e4600e8e0a9690b not found: ID does not exist"
Sep 15 07:48:29.939964 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.939906    2571 scope.go:117] "RemoveContainer" containerID="cf3f6c8128999ad212b987fd45e888de8f8d61e161e971ebdef341af564d40f6"
Sep 15 07:48:29.951641 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.951620    2571 scope.go:117] "RemoveContainer" containerID="cf3f6c8128999ad212b987fd45e888de8f8d61e161e971ebdef341af564d40f6"
Sep 15 07:48:29.951966 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:48:29.951943    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf3f6c8128999ad212b987fd45e888de8f8d61e161e971ebdef341af564d40f6\": container with ID starting with cf3f6c8128999ad212b987fd45e888de8f8d61e161e971ebdef341af564d40f6 not found: ID does not exist" containerID="cf3f6c8128999ad212b987fd45e888de8f8d61e161e971ebdef341af564d40f6"
Sep 15 07:48:29.952046 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.951973    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3f6c8128999ad212b987fd45e888de8f8d61e161e971ebdef341af564d40f6"} err="failed to get container status \"cf3f6c8128999ad212b987fd45e888de8f8d61e161e971ebdef341af564d40f6\": rpc error: code = NotFound desc = could not find container \"cf3f6c8128999ad212b987fd45e888de8f8d61e161e971ebdef341af564d40f6\": container with ID starting with cf3f6c8128999ad212b987fd45e888de8f8d61e161e971ebdef341af564d40f6 not found: ID does not exist"
Sep 15 07:48:29.956833 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.956810    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"]
Sep 15 07:48:29.964646 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.964622    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/simulator-w0-2735c5-kserve-6fcdb657c6-v5sl7"]
Sep 15 07:48:29.977221 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.977167    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"]
Sep 15 07:48:29.979727 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.979706    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/distinct-2-w0-2735c5-kserve-67b4b5c75f-6tvqp"]
Sep 15 07:48:29.993212 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.993188    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57z9l\" (UniqueName: \"kubernetes.io/projected/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-kube-api-access-57z9l\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:48:29.993331 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.993217    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-tls-certs\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:48:29.993331 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.993235    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-model-cache\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:48:29.993331 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.993246    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-dshm\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:48:29.993331 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.993271    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dff30f35-72cb-4fe8-b87d-8bc20401fd8d-model-cache\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:48:29.993331 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.993284    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4190df-9881-42fc-bda5-b73881600bf6-tls-certs\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:48:29.993331 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.993295    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4190df-9881-42fc-bda5-b73881600bf6-dshm\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:48:29.993331 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:29.993307    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gkg8\" (UniqueName: \"kubernetes.io/projected/6c4190df-9881-42fc-bda5-b73881600bf6-kube-api-access-7gkg8\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:48:31.910839 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:31.910807    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c4190df-9881-42fc-bda5-b73881600bf6" path="/var/lib/kubelet/pods/6c4190df-9881-42fc-bda5-b73881600bf6/volumes"
Sep 15 07:48:31.911297 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:31.911281    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dff30f35-72cb-4fe8-b87d-8bc20401fd8d" path="/var/lib/kubelet/pods/dff30f35-72cb-4fe8-b87d-8bc20401fd8d/volumes"
Sep 15 07:48:40.994138 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:40.994106    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj"]
Sep 15 07:48:40.994599 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:40.994584    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff30f35-72cb-4fe8-b87d-8bc20401fd8d" containerName="main"
Sep 15 07:48:40.994651 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:40.994603    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff30f35-72cb-4fe8-b87d-8bc20401fd8d" containerName="main"
Sep 15 07:48:40.994651 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:40.994619    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c4190df-9881-42fc-bda5-b73881600bf6" containerName="main"
Sep 15 07:48:40.994651 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:40.994628    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c4190df-9881-42fc-bda5-b73881600bf6" containerName="main"
Sep 15 07:48:40.994651 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:40.994643    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6c1f080-389f-4ab8-abd7-e62804e7302f" containerName="payload-pre-processing"
Sep 15 07:48:40.994651 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:40.994648    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6c1f080-389f-4ab8-abd7-e62804e7302f" containerName="payload-pre-processing"
Sep 15 07:48:40.994802 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:40.994727    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="dff30f35-72cb-4fe8-b87d-8bc20401fd8d" containerName="main"
Sep 15 07:48:40.994802 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:40.994738    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6c1f080-389f-4ab8-abd7-e62804e7302f" containerName="payload-pre-processing"
Sep 15 07:48:40.994802 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:40.994744    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c4190df-9881-42fc-bda5-b73881600bf6" containerName="main"
Sep 15 07:48:41.000190 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.000169    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj"
Sep 15 07:48:41.002726 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.002705    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1755f7\""
Sep 15 07:48:41.002726 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.002708    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-1755f7-dockercfg-5kvpx\""
Sep 15 07:48:41.006304 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.006280    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8"]
Sep 15 07:48:41.010662 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.010627    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj"]
Sep 15 07:48:41.010778 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.010748    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8"
Sep 15 07:48:41.021881 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.021852    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8"]
Sep 15 07:48:41.107788 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.107751    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhrqj\" (UniqueName: \"kubernetes.io/projected/7b2502c9-e9d7-4ecf-b431-cd82e5a793fa-kube-api-access-mhrqj\") pod \"payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8\" (UID: \"7b2502c9-e9d7-4ecf-b431-cd82e5a793fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8"
Sep 15 07:48:41.107987 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.107806    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2502c9-e9d7-4ecf-b431-cd82e5a793fa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8\" (UID: \"7b2502c9-e9d7-4ecf-b431-cd82e5a793fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8"
Sep 15 07:48:41.107987 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.107828    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwc9w\" (UniqueName: \"kubernetes.io/projected/4a09614a-200b-4d0d-bc0f-59b1313f3c62-kube-api-access-dwc9w\") pod \"payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj\" (UID: \"4a09614a-200b-4d0d-bc0f-59b1313f3c62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj"
Sep 15 07:48:41.107987 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.107947    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a09614a-200b-4d0d-bc0f-59b1313f3c62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj\" (UID: \"4a09614a-200b-4d0d-bc0f-59b1313f3c62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj"
Sep 15 07:48:41.209279 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.209219    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhrqj\" (UniqueName: \"kubernetes.io/projected/7b2502c9-e9d7-4ecf-b431-cd82e5a793fa-kube-api-access-mhrqj\") pod \"payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8\" (UID: \"7b2502c9-e9d7-4ecf-b431-cd82e5a793fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8"
Sep 15 07:48:41.209462 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.209310    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2502c9-e9d7-4ecf-b431-cd82e5a793fa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8\" (UID: \"7b2502c9-e9d7-4ecf-b431-cd82e5a793fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8"
Sep 15 07:48:41.209462 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.209329    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwc9w\" (UniqueName: \"kubernetes.io/projected/4a09614a-200b-4d0d-bc0f-59b1313f3c62-kube-api-access-dwc9w\") pod \"payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj\" (UID: \"4a09614a-200b-4d0d-bc0f-59b1313f3c62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj"
Sep 15 07:48:41.209462 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.209384    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a09614a-200b-4d0d-bc0f-59b1313f3c62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj\" (UID: \"4a09614a-200b-4d0d-bc0f-59b1313f3c62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj"
Sep 15 07:48:41.210107 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.210087    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a09614a-200b-4d0d-bc0f-59b1313f3c62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj\" (UID: \"4a09614a-200b-4d0d-bc0f-59b1313f3c62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj"
Sep 15 07:48:41.210150 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.210091    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2502c9-e9d7-4ecf-b431-cd82e5a793fa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8\" (UID: \"7b2502c9-e9d7-4ecf-b431-cd82e5a793fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8"
Sep 15 07:48:41.218045 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.218018    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwc9w\" (UniqueName: \"kubernetes.io/projected/4a09614a-200b-4d0d-bc0f-59b1313f3c62-kube-api-access-dwc9w\") pod \"payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj\" (UID: \"4a09614a-200b-4d0d-bc0f-59b1313f3c62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj"
Sep 15 07:48:41.218142 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.218098    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhrqj\" (UniqueName: \"kubernetes.io/projected/7b2502c9-e9d7-4ecf-b431-cd82e5a793fa-kube-api-access-mhrqj\") pod \"payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8\" (UID: \"7b2502c9-e9d7-4ecf-b431-cd82e5a793fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8"
Sep 15 07:48:41.313985 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.313903    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj"
Sep 15 07:48:41.324006 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.323980    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8"
Sep 15 07:48:41.595578 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.595537    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj"]
Sep 15 07:48:41.598736 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:48:41.598707    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a09614a_200b_4d0d_bc0f_59b1313f3c62.slice/crio-4195ac1cb8599fbeccadd5bfab65596c7a161cc83f7ebb444b16415513fd7afe WatchSource:0}: Error finding container 4195ac1cb8599fbeccadd5bfab65596c7a161cc83f7ebb444b16415513fd7afe: Status 404 returned error can't find the container with id 4195ac1cb8599fbeccadd5bfab65596c7a161cc83f7ebb444b16415513fd7afe
Sep 15 07:48:41.605891 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:41.605818    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8"]
Sep 15 07:48:41.608184 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:48:41.608159    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2502c9_e9d7_4ecf_b431_cd82e5a793fa.slice/crio-eb4bbf6b8b352d70255dd38092de57869eb1514754046b0595350cade066a955 WatchSource:0}: Error finding container eb4bbf6b8b352d70255dd38092de57869eb1514754046b0595350cade066a955: Status 404 returned error can't find the container with id eb4bbf6b8b352d70255dd38092de57869eb1514754046b0595350cade066a955
Sep 15 07:48:42.000504 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:42.000464    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8" event={"ID":"7b2502c9-e9d7-4ecf-b431-cd82e5a793fa","Type":"ContainerStarted","Data":"412f1288065040aaca967775999a62faa55ec3334ef7ea3baf6c5e0651c4c329"}
Sep 15 07:48:42.001000 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:42.000538    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8"
Sep 15 07:48:42.001000 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:42.000563    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8" event={"ID":"7b2502c9-e9d7-4ecf-b431-cd82e5a793fa","Type":"ContainerStarted","Data":"eb4bbf6b8b352d70255dd38092de57869eb1514754046b0595350cade066a955"}
Sep 15 07:48:42.002109 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:42.002086    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj" event={"ID":"4a09614a-200b-4d0d-bc0f-59b1313f3c62","Type":"ContainerStarted","Data":"24d9fc7f55ed3f3cb43f671dbe920bab245f95f09f0525d3d912bef01750a652"}
Sep 15 07:48:42.002202 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:42.002129    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj" event={"ID":"4a09614a-200b-4d0d-bc0f-59b1313f3c62","Type":"ContainerStarted","Data":"4195ac1cb8599fbeccadd5bfab65596c7a161cc83f7ebb444b16415513fd7afe"}
Sep 15 07:48:42.002202 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:42.002169    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj"
Sep 15 07:48:42.016905 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:42.016824    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8" podStartSLOduration=2.01680785 podStartE2EDuration="2.01680785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:48:42.016443067 +0000 UTC m=+1826.563013197" watchObservedRunningTime="2026-09-15 07:48:42.01680785 +0000 UTC m=+1826.563377950"
Sep 15 07:48:42.036173 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:42.036116    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj" podStartSLOduration=2.036097662 podStartE2EDuration="2.036097662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:48:42.031904055 +0000 UTC m=+1826.578474173" watchObservedRunningTime="2026-09-15 07:48:42.036097662 +0000 UTC m=+1826.582667763"
Sep 15 07:48:53.011435 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:53.011403    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8"
Sep 15 07:48:53.011950 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:48:53.011479    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj"
Sep 15 07:49:06.760979 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:06.760943    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ef86b-twvw4"]
Sep 15 07:49:06.764785 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:06.764765    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ef86b-twvw4"
Sep 15 07:49:06.767276 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:06.767238    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 07:49:06.767384 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:06.767301    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8jpwk\""
Sep 15 07:49:06.771715 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:06.771692    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ef86b-twvw4"]
Sep 15 07:49:06.854348 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:06.854307    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b53d9786-e345-4c0e-95c1-3980704dbc36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4ef86b-twvw4\" (UID: \"b53d9786-e345-4c0e-95c1-3980704dbc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ef86b-twvw4"
Sep 15 07:49:06.955121 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:06.955068    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b53d9786-e345-4c0e-95c1-3980704dbc36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4ef86b-twvw4\" (UID: \"b53d9786-e345-4c0e-95c1-3980704dbc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ef86b-twvw4"
Sep 15 07:49:06.955792 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:06.955763    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b53d9786-e345-4c0e-95c1-3980704dbc36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4ef86b-twvw4\" (UID: \"b53d9786-e345-4c0e-95c1-3980704dbc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ef86b-twvw4"
Sep 15 07:49:07.077195 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:07.077105    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ef86b-twvw4"
Sep 15 07:49:07.360685 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:07.360635    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ef86b-twvw4"]
Sep 15 07:49:08.161040 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:08.161003    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ef86b-twvw4" event={"ID":"b53d9786-e345-4c0e-95c1-3980704dbc36","Type":"ContainerStarted","Data":"3380264c958dc6bdb9d1948ba0ac18cdebdec15e464886ec70aa8b1687e02dc2"}
Sep 15 07:49:08.161464 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:08.161057    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ef86b-twvw4" event={"ID":"b53d9786-e345-4c0e-95c1-3980704dbc36","Type":"ContainerStarted","Data":"aa82838362bd1e6c8cfcd15d1d10f1d212a81541c057ea9b3129f9542d3b21ea"}
Sep 15 07:49:08.178531 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:08.178474    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ef86b-twvw4" podStartSLOduration=2.178456982 podStartE2EDuration="2.178456982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:49:08.176804308 +0000 UTC m=+1852.723374424" watchObservedRunningTime="2026-09-15 07:49:08.178456982 +0000 UTC m=+1852.725027082"
Sep 15 07:49:09.168870 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:09.168836    2571 generic.go:356] "Generic (PLEG): container finished" podID="b53d9786-e345-4c0e-95c1-3980704dbc36" containerID="3380264c958dc6bdb9d1948ba0ac18cdebdec15e464886ec70aa8b1687e02dc2" exitCode=0
Sep 15 07:49:09.169280 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:09.168880    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ef86b-twvw4" event={"ID":"b53d9786-e345-4c0e-95c1-3980704dbc36","Type":"ContainerDied","Data":"3380264c958dc6bdb9d1948ba0ac18cdebdec15e464886ec70aa8b1687e02dc2"}
Sep 15 07:49:10.320022 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:10.319990    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ef86b-twvw4"
Sep 15 07:49:10.392291 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:10.392238    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b53d9786-e345-4c0e-95c1-3980704dbc36-maas-api-service-ca\") pod \"b53d9786-e345-4c0e-95c1-3980704dbc36\" (UID: \"b53d9786-e345-4c0e-95c1-3980704dbc36\") "
Sep 15 07:49:10.392591 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:10.392567    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b53d9786-e345-4c0e-95c1-3980704dbc36-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b53d9786-e345-4c0e-95c1-3980704dbc36" (UID: "b53d9786-e345-4c0e-95c1-3980704dbc36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:49:10.493058 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:10.493026    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b53d9786-e345-4c0e-95c1-3980704dbc36-maas-api-service-ca\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:49:11.181931 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:11.181893    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ef86b-twvw4"
Sep 15 07:49:11.182093 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:11.181943    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ef86b-twvw4" event={"ID":"b53d9786-e345-4c0e-95c1-3980704dbc36","Type":"ContainerDied","Data":"aa82838362bd1e6c8cfcd15d1d10f1d212a81541c057ea9b3129f9542d3b21ea"}
Sep 15 07:49:11.182093 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:11.181994    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa82838362bd1e6c8cfcd15d1d10f1d212a81541c057ea9b3129f9542d3b21ea"
Sep 15 07:49:16.116351 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:16.116318    2571 scope.go:117] "RemoveContainer" containerID="1d1157322e45d6c6fc4695abbd4576c8031a62fa9ea3c6d4b45b2f1078531604"
Sep 15 07:49:16.125676 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:16.125655    2571 scope.go:117] "RemoveContainer" containerID="ae429ffe0e5fd9fae0c1f590f82ea356751e8b1ec94287c4a805d48f52f3085b"
Sep 15 07:49:22.971649 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:22.971611    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ef86b-twvw4"]
Sep 15 07:49:22.973911 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:22.973885    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ef86b-twvw4"]
Sep 15 07:49:23.910361 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:23.910330    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b53d9786-e345-4c0e-95c1-3980704dbc36" path="/var/lib/kubelet/pods/b53d9786-e345-4c0e-95c1-3980704dbc36/volumes"
Sep 15 07:49:46.270673 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:46.270637    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8"]
Sep 15 07:49:46.271132 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:46.270934    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8" podUID="7b2502c9-e9d7-4ecf-b431-cd82e5a793fa" containerName="payload-processing" containerID="cri-o://412f1288065040aaca967775999a62faa55ec3334ef7ea3baf6c5e0651c4c329" gracePeriod=30
Sep 15 07:49:46.279459 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:46.279241    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj"]
Sep 15 07:49:46.279575 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:46.279480    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj" podUID="4a09614a-200b-4d0d-bc0f-59b1313f3c62" containerName="payload-pre-processing" containerID="cri-o://24d9fc7f55ed3f3cb43f671dbe920bab245f95f09f0525d3d912bef01750a652" gracePeriod=30
Sep 15 07:49:46.392332 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:46.392295    2571 generic.go:356] "Generic (PLEG): container finished" podID="7b2502c9-e9d7-4ecf-b431-cd82e5a793fa" containerID="412f1288065040aaca967775999a62faa55ec3334ef7ea3baf6c5e0651c4c329" exitCode=0
Sep 15 07:49:46.392454 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:46.392366    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8" event={"ID":"7b2502c9-e9d7-4ecf-b431-cd82e5a793fa","Type":"ContainerDied","Data":"412f1288065040aaca967775999a62faa55ec3334ef7ea3baf6c5e0651c4c329"}
Sep 15 07:49:46.554135 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:46.554110    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8"
Sep 15 07:49:46.557067 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:46.557048    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj"
Sep 15 07:49:46.655437 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:46.655400    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwc9w\" (UniqueName: \"kubernetes.io/projected/4a09614a-200b-4d0d-bc0f-59b1313f3c62-kube-api-access-dwc9w\") pod \"4a09614a-200b-4d0d-bc0f-59b1313f3c62\" (UID: \"4a09614a-200b-4d0d-bc0f-59b1313f3c62\") "
Sep 15 07:49:46.655609 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:46.655488    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2502c9-e9d7-4ecf-b431-cd82e5a793fa-plugins-config-volume\") pod \"7b2502c9-e9d7-4ecf-b431-cd82e5a793fa\" (UID: \"7b2502c9-e9d7-4ecf-b431-cd82e5a793fa\") "
Sep 15 07:49:46.655609 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:46.655520    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhrqj\" (UniqueName: \"kubernetes.io/projected/7b2502c9-e9d7-4ecf-b431-cd82e5a793fa-kube-api-access-mhrqj\") pod \"7b2502c9-e9d7-4ecf-b431-cd82e5a793fa\" (UID: \"7b2502c9-e9d7-4ecf-b431-cd82e5a793fa\") "
Sep 15 07:49:46.655609 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:46.655550    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a09614a-200b-4d0d-bc0f-59b1313f3c62-plugins-config-volume\") pod \"4a09614a-200b-4d0d-bc0f-59b1313f3c62\" (UID: \"4a09614a-200b-4d0d-bc0f-59b1313f3c62\") "
Sep 15 07:49:46.655893 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:46.655867    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b2502c9-e9d7-4ecf-b431-cd82e5a793fa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7b2502c9-e9d7-4ecf-b431-cd82e5a793fa" (UID: "7b2502c9-e9d7-4ecf-b431-cd82e5a793fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:49:46.655958 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:46.655938    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a09614a-200b-4d0d-bc0f-59b1313f3c62-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4a09614a-200b-4d0d-bc0f-59b1313f3c62" (UID: "4a09614a-200b-4d0d-bc0f-59b1313f3c62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:49:46.656012 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:46.655993    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2502c9-e9d7-4ecf-b431-cd82e5a793fa-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:49:46.657665 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:46.657634    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a09614a-200b-4d0d-bc0f-59b1313f3c62-kube-api-access-dwc9w" (OuterVolumeSpecName: "kube-api-access-dwc9w") pod "4a09614a-200b-4d0d-bc0f-59b1313f3c62" (UID: "4a09614a-200b-4d0d-bc0f-59b1313f3c62"). InnerVolumeSpecName "kube-api-access-dwc9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:49:46.657766 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:46.657674    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2502c9-e9d7-4ecf-b431-cd82e5a793fa-kube-api-access-mhrqj" (OuterVolumeSpecName: "kube-api-access-mhrqj") pod "7b2502c9-e9d7-4ecf-b431-cd82e5a793fa" (UID: "7b2502c9-e9d7-4ecf-b431-cd82e5a793fa"). InnerVolumeSpecName "kube-api-access-mhrqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:49:46.757303 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:46.757236    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwc9w\" (UniqueName: \"kubernetes.io/projected/4a09614a-200b-4d0d-bc0f-59b1313f3c62-kube-api-access-dwc9w\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:49:46.757303 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:46.757305    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhrqj\" (UniqueName: \"kubernetes.io/projected/7b2502c9-e9d7-4ecf-b431-cd82e5a793fa-kube-api-access-mhrqj\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:49:46.757486 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:46.757320    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a09614a-200b-4d0d-bc0f-59b1313f3c62-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:49:47.402938 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:47.402901    2571 generic.go:356] "Generic (PLEG): container finished" podID="4a09614a-200b-4d0d-bc0f-59b1313f3c62" containerID="24d9fc7f55ed3f3cb43f671dbe920bab245f95f09f0525d3d912bef01750a652" exitCode=0
Sep 15 07:49:47.403423 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:47.402949    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj"
Sep 15 07:49:47.403423 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:47.402982    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj" event={"ID":"4a09614a-200b-4d0d-bc0f-59b1313f3c62","Type":"ContainerDied","Data":"24d9fc7f55ed3f3cb43f671dbe920bab245f95f09f0525d3d912bef01750a652"}
Sep 15 07:49:47.403423 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:47.403035    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj" event={"ID":"4a09614a-200b-4d0d-bc0f-59b1313f3c62","Type":"ContainerDied","Data":"4195ac1cb8599fbeccadd5bfab65596c7a161cc83f7ebb444b16415513fd7afe"}
Sep 15 07:49:47.403423 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:47.403056    2571 scope.go:117] "RemoveContainer" containerID="24d9fc7f55ed3f3cb43f671dbe920bab245f95f09f0525d3d912bef01750a652"
Sep 15 07:49:47.404770 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:47.404751    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8" event={"ID":"7b2502c9-e9d7-4ecf-b431-cd82e5a793fa","Type":"ContainerDied","Data":"eb4bbf6b8b352d70255dd38092de57869eb1514754046b0595350cade066a955"}
Sep 15 07:49:47.404877 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:47.404770    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8"
Sep 15 07:49:47.414053 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:47.414002    2571 scope.go:117] "RemoveContainer" containerID="24d9fc7f55ed3f3cb43f671dbe920bab245f95f09f0525d3d912bef01750a652"
Sep 15 07:49:47.414386 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:49:47.414357    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d9fc7f55ed3f3cb43f671dbe920bab245f95f09f0525d3d912bef01750a652\": container with ID starting with 24d9fc7f55ed3f3cb43f671dbe920bab245f95f09f0525d3d912bef01750a652 not found: ID does not exist" containerID="24d9fc7f55ed3f3cb43f671dbe920bab245f95f09f0525d3d912bef01750a652"
Sep 15 07:49:47.414491 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:47.414395    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d9fc7f55ed3f3cb43f671dbe920bab245f95f09f0525d3d912bef01750a652"} err="failed to get container status \"24d9fc7f55ed3f3cb43f671dbe920bab245f95f09f0525d3d912bef01750a652\": rpc error: code = NotFound desc = could not find container \"24d9fc7f55ed3f3cb43f671dbe920bab245f95f09f0525d3d912bef01750a652\": container with ID starting with 24d9fc7f55ed3f3cb43f671dbe920bab245f95f09f0525d3d912bef01750a652 not found: ID does not exist"
Sep 15 07:49:47.414491 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:47.414416    2571 scope.go:117] "RemoveContainer" containerID="412f1288065040aaca967775999a62faa55ec3334ef7ea3baf6c5e0651c4c329"
Sep 15 07:49:47.428594 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:47.428571    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj"]
Sep 15 07:49:47.432972 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:47.432950    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1755f7-6cfd449466-c5mgj"]
Sep 15 07:49:47.444910 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:47.444888    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8"]
Sep 15 07:49:47.449354 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:47.449335    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1755f7-785bcdd875-87qk8"]
Sep 15 07:49:47.910384 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:47.910351    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a09614a-200b-4d0d-bc0f-59b1313f3c62" path="/var/lib/kubelet/pods/4a09614a-200b-4d0d-bc0f-59b1313f3c62/volumes"
Sep 15 07:49:47.910724 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:49:47.910711    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b2502c9-e9d7-4ecf-b431-cd82e5a793fa" path="/var/lib/kubelet/pods/7b2502c9-e9d7-4ecf-b431-cd82e5a793fa/volumes"
Sep 15 07:50:04.095752 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:04.095695    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3a4c9f-29824305-q84rz"]
Sep 15 07:50:04.097869 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:04.097843    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3a4c9f-29824305-q84rz"]
Sep 15 07:50:05.909904 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:05.909874    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d4810f8-42e1-438d-94f1-83ebb42caaa7" path="/var/lib/kubelet/pods/0d4810f8-42e1-438d-94f1-83ebb42caaa7/volumes"
Sep 15 07:50:22.228639 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.228604    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2"]
Sep 15 07:50:22.229106 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.228896    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2" podUID="c672cec7-2811-4072-961f-9dd921e5aef5" containerName="payload-processing" containerID="cri-o://26c5bcbc02c4eeb1989a9e4621b3824e8d6faea186665d5348b0bd7789865700" gracePeriod=30
Sep 15 07:50:22.235934 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.235906    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd"]
Sep 15 07:50:22.236832 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.236795    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd" podUID="60f1cb03-c1e7-4820-934d-62efeb27d9ea" containerName="payload-pre-processing" containerID="cri-o://284a06d0d4fdb049655702c104c7846605edbec03291d484be6ea8011bf63a0d" gracePeriod=30
Sep 15 07:50:22.520380 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.520357    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2"
Sep 15 07:50:22.523086 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.523068    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd"
Sep 15 07:50:22.606126 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.606090    2571 generic.go:356] "Generic (PLEG): container finished" podID="60f1cb03-c1e7-4820-934d-62efeb27d9ea" containerID="284a06d0d4fdb049655702c104c7846605edbec03291d484be6ea8011bf63a0d" exitCode=0
Sep 15 07:50:22.606346 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.606142    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd"
Sep 15 07:50:22.606346 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.606182    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd" event={"ID":"60f1cb03-c1e7-4820-934d-62efeb27d9ea","Type":"ContainerDied","Data":"284a06d0d4fdb049655702c104c7846605edbec03291d484be6ea8011bf63a0d"}
Sep 15 07:50:22.606346 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.606235    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd" event={"ID":"60f1cb03-c1e7-4820-934d-62efeb27d9ea","Type":"ContainerDied","Data":"ddb9fddfb342dd878b26fecfe2c947010b70c913d573e60896d52d78820d736c"}
Sep 15 07:50:22.606346 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.606283    2571 scope.go:117] "RemoveContainer" containerID="284a06d0d4fdb049655702c104c7846605edbec03291d484be6ea8011bf63a0d"
Sep 15 07:50:22.607583 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.607558    2571 generic.go:356] "Generic (PLEG): container finished" podID="c672cec7-2811-4072-961f-9dd921e5aef5" containerID="26c5bcbc02c4eeb1989a9e4621b3824e8d6faea186665d5348b0bd7789865700" exitCode=0
Sep 15 07:50:22.607688 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.607587    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2" event={"ID":"c672cec7-2811-4072-961f-9dd921e5aef5","Type":"ContainerDied","Data":"26c5bcbc02c4eeb1989a9e4621b3824e8d6faea186665d5348b0bd7789865700"}
Sep 15 07:50:22.607688 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.607623    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2"
Sep 15 07:50:22.607688 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.607635    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2" event={"ID":"c672cec7-2811-4072-961f-9dd921e5aef5","Type":"ContainerDied","Data":"b9a78b339a353a99290c70570c9c6ca3e803b3996a8160e7d538e2f02aea9a58"}
Sep 15 07:50:22.615604 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.615583    2571 scope.go:117] "RemoveContainer" containerID="284a06d0d4fdb049655702c104c7846605edbec03291d484be6ea8011bf63a0d"
Sep 15 07:50:22.615856 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:50:22.615840    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"284a06d0d4fdb049655702c104c7846605edbec03291d484be6ea8011bf63a0d\": container with ID starting with 284a06d0d4fdb049655702c104c7846605edbec03291d484be6ea8011bf63a0d not found: ID does not exist" containerID="284a06d0d4fdb049655702c104c7846605edbec03291d484be6ea8011bf63a0d"
Sep 15 07:50:22.615900 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.615864    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"284a06d0d4fdb049655702c104c7846605edbec03291d484be6ea8011bf63a0d"} err="failed to get container status \"284a06d0d4fdb049655702c104c7846605edbec03291d484be6ea8011bf63a0d\": rpc error: code = NotFound desc = could not find container \"284a06d0d4fdb049655702c104c7846605edbec03291d484be6ea8011bf63a0d\": container with ID starting with 284a06d0d4fdb049655702c104c7846605edbec03291d484be6ea8011bf63a0d not found: ID does not exist"
Sep 15 07:50:22.615900 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.615879    2571 scope.go:117] "RemoveContainer" containerID="26c5bcbc02c4eeb1989a9e4621b3824e8d6faea186665d5348b0bd7789865700"
Sep 15 07:50:22.622161 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.622135    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l592s\" (UniqueName: \"kubernetes.io/projected/c672cec7-2811-4072-961f-9dd921e5aef5-kube-api-access-l592s\") pod \"c672cec7-2811-4072-961f-9dd921e5aef5\" (UID: \"c672cec7-2811-4072-961f-9dd921e5aef5\") "
Sep 15 07:50:22.622245 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.622202    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c672cec7-2811-4072-961f-9dd921e5aef5-plugins-config-volume\") pod \"c672cec7-2811-4072-961f-9dd921e5aef5\" (UID: \"c672cec7-2811-4072-961f-9dd921e5aef5\") "
Sep 15 07:50:22.622389 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.622374    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f1cb03-c1e7-4820-934d-62efeb27d9ea-plugins-config-volume\") pod \"60f1cb03-c1e7-4820-934d-62efeb27d9ea\" (UID: \"60f1cb03-c1e7-4820-934d-62efeb27d9ea\") "
Sep 15 07:50:22.622443 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.622425    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk2kb\" (UniqueName: \"kubernetes.io/projected/60f1cb03-c1e7-4820-934d-62efeb27d9ea-kube-api-access-gk2kb\") pod \"60f1cb03-c1e7-4820-934d-62efeb27d9ea\" (UID: \"60f1cb03-c1e7-4820-934d-62efeb27d9ea\") "
Sep 15 07:50:22.622587 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.622556    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c672cec7-2811-4072-961f-9dd921e5aef5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c672cec7-2811-4072-961f-9dd921e5aef5" (UID: "c672cec7-2811-4072-961f-9dd921e5aef5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:50:22.622809 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.622776    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c672cec7-2811-4072-961f-9dd921e5aef5-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:50:22.622809 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.622773    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f1cb03-c1e7-4820-934d-62efeb27d9ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "60f1cb03-c1e7-4820-934d-62efeb27d9ea" (UID: "60f1cb03-c1e7-4820-934d-62efeb27d9ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:50:22.624223 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.624196    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c672cec7-2811-4072-961f-9dd921e5aef5-kube-api-access-l592s" (OuterVolumeSpecName: "kube-api-access-l592s") pod "c672cec7-2811-4072-961f-9dd921e5aef5" (UID: "c672cec7-2811-4072-961f-9dd921e5aef5"). InnerVolumeSpecName "kube-api-access-l592s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:50:22.624488 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.624464    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f1cb03-c1e7-4820-934d-62efeb27d9ea-kube-api-access-gk2kb" (OuterVolumeSpecName: "kube-api-access-gk2kb") pod "60f1cb03-c1e7-4820-934d-62efeb27d9ea" (UID: "60f1cb03-c1e7-4820-934d-62efeb27d9ea"). InnerVolumeSpecName "kube-api-access-gk2kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:50:22.624822 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.624803    2571 scope.go:117] "RemoveContainer" containerID="26c5bcbc02c4eeb1989a9e4621b3824e8d6faea186665d5348b0bd7789865700"
Sep 15 07:50:22.625100 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:50:22.625078    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c5bcbc02c4eeb1989a9e4621b3824e8d6faea186665d5348b0bd7789865700\": container with ID starting with 26c5bcbc02c4eeb1989a9e4621b3824e8d6faea186665d5348b0bd7789865700 not found: ID does not exist" containerID="26c5bcbc02c4eeb1989a9e4621b3824e8d6faea186665d5348b0bd7789865700"
Sep 15 07:50:22.625142 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.625109    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c5bcbc02c4eeb1989a9e4621b3824e8d6faea186665d5348b0bd7789865700"} err="failed to get container status \"26c5bcbc02c4eeb1989a9e4621b3824e8d6faea186665d5348b0bd7789865700\": rpc error: code = NotFound desc = could not find container \"26c5bcbc02c4eeb1989a9e4621b3824e8d6faea186665d5348b0bd7789865700\": container with ID starting with 26c5bcbc02c4eeb1989a9e4621b3824e8d6faea186665d5348b0bd7789865700 not found: ID does not exist"
Sep 15 07:50:22.723656 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.723618    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk2kb\" (UniqueName: \"kubernetes.io/projected/60f1cb03-c1e7-4820-934d-62efeb27d9ea-kube-api-access-gk2kb\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:50:22.723656 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.723646    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l592s\" (UniqueName: \"kubernetes.io/projected/c672cec7-2811-4072-961f-9dd921e5aef5-kube-api-access-l592s\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:50:22.723656 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.723655    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f1cb03-c1e7-4820-934d-62efeb27d9ea-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:50:22.930576 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.930540    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2"]
Sep 15 07:50:22.934036 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.934014    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-27b9d8-6dcfcf888c-dqkb2"]
Sep 15 07:50:22.944984 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.944954    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd"]
Sep 15 07:50:22.950062 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:22.950038    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-27b9d8-f8c7f74c8-gqbfd"]
Sep 15 07:50:23.909964 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:23.909926    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60f1cb03-c1e7-4820-934d-62efeb27d9ea" path="/var/lib/kubelet/pods/60f1cb03-c1e7-4820-934d-62efeb27d9ea/volumes"
Sep 15 07:50:23.910347 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:23.910325    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c672cec7-2811-4072-961f-9dd921e5aef5" path="/var/lib/kubelet/pods/c672cec7-2811-4072-961f-9dd921e5aef5/volumes"
Sep 15 07:50:50.310667 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:50.310632    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"]
Sep 15 07:50:50.331429 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:50:50.331394    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"]
Sep 15 07:51:01.168960 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.168922    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26"]
Sep 15 07:51:01.169518 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.169176    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26" podUID="827d14fe-b570-49ca-8600-778f1146b956" containerName="payload-processing" containerID="cri-o://74ddd16112e4aa739bcdde7cf119e269ac64f2ba669a2f9489fb48efc5ee31b6" gracePeriod=30
Sep 15 07:51:01.175159 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.175128    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr"]
Sep 15 07:51:01.175605 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.175539    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr" podUID="3e704c76-3c4d-4e0d-83b4-c0e9115f33c1" containerName="payload-pre-processing" containerID="cri-o://ccbc8d073f845e3dfd86da331719a0964903fbee4338bf95c37df81df291a1de" gracePeriod=30
Sep 15 07:51:01.447331 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.447306    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26"
Sep 15 07:51:01.453998 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.453976    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr"
Sep 15 07:51:01.496187 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.496155    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kc9v\" (UniqueName: \"kubernetes.io/projected/3e704c76-3c4d-4e0d-83b4-c0e9115f33c1-kube-api-access-4kc9v\") pod \"3e704c76-3c4d-4e0d-83b4-c0e9115f33c1\" (UID: \"3e704c76-3c4d-4e0d-83b4-c0e9115f33c1\") "
Sep 15 07:51:01.496380 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.496204    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5ms5\" (UniqueName: \"kubernetes.io/projected/827d14fe-b570-49ca-8600-778f1146b956-kube-api-access-p5ms5\") pod \"827d14fe-b570-49ca-8600-778f1146b956\" (UID: \"827d14fe-b570-49ca-8600-778f1146b956\") "
Sep 15 07:51:01.496380 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.496297    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/827d14fe-b570-49ca-8600-778f1146b956-plugins-config-volume\") pod \"827d14fe-b570-49ca-8600-778f1146b956\" (UID: \"827d14fe-b570-49ca-8600-778f1146b956\") "
Sep 15 07:51:01.496380 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.496326    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e704c76-3c4d-4e0d-83b4-c0e9115f33c1-plugins-config-volume\") pod \"3e704c76-3c4d-4e0d-83b4-c0e9115f33c1\" (UID: \"3e704c76-3c4d-4e0d-83b4-c0e9115f33c1\") "
Sep 15 07:51:01.496739 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.496710    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e704c76-3c4d-4e0d-83b4-c0e9115f33c1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e704c76-3c4d-4e0d-83b4-c0e9115f33c1" (UID: "3e704c76-3c4d-4e0d-83b4-c0e9115f33c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:51:01.496870 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.496738    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/827d14fe-b570-49ca-8600-778f1146b956-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "827d14fe-b570-49ca-8600-778f1146b956" (UID: "827d14fe-b570-49ca-8600-778f1146b956"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:51:01.498401 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.498378    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e704c76-3c4d-4e0d-83b4-c0e9115f33c1-kube-api-access-4kc9v" (OuterVolumeSpecName: "kube-api-access-4kc9v") pod "3e704c76-3c4d-4e0d-83b4-c0e9115f33c1" (UID: "3e704c76-3c4d-4e0d-83b4-c0e9115f33c1"). InnerVolumeSpecName "kube-api-access-4kc9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:51:01.498519 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.498402    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827d14fe-b570-49ca-8600-778f1146b956-kube-api-access-p5ms5" (OuterVolumeSpecName: "kube-api-access-p5ms5") pod "827d14fe-b570-49ca-8600-778f1146b956" (UID: "827d14fe-b570-49ca-8600-778f1146b956"). InnerVolumeSpecName "kube-api-access-p5ms5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:51:01.597438 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.597400    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kc9v\" (UniqueName: \"kubernetes.io/projected/3e704c76-3c4d-4e0d-83b4-c0e9115f33c1-kube-api-access-4kc9v\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:01.597438 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.597433    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5ms5\" (UniqueName: \"kubernetes.io/projected/827d14fe-b570-49ca-8600-778f1146b956-kube-api-access-p5ms5\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:01.597438 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.597442    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/827d14fe-b570-49ca-8600-778f1146b956-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:01.597691 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.597450    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e704c76-3c4d-4e0d-83b4-c0e9115f33c1-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:01.830669 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.830632    2571 generic.go:356] "Generic (PLEG): container finished" podID="827d14fe-b570-49ca-8600-778f1146b956" containerID="74ddd16112e4aa739bcdde7cf119e269ac64f2ba669a2f9489fb48efc5ee31b6" exitCode=0
Sep 15 07:51:01.830669 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.830674    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26"
Sep 15 07:51:01.830912 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.830715    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26" event={"ID":"827d14fe-b570-49ca-8600-778f1146b956","Type":"ContainerDied","Data":"74ddd16112e4aa739bcdde7cf119e269ac64f2ba669a2f9489fb48efc5ee31b6"}
Sep 15 07:51:01.830912 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.830769    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26" event={"ID":"827d14fe-b570-49ca-8600-778f1146b956","Type":"ContainerDied","Data":"ae063872b5a4541bd5d45d5c9ea9366ec1a773d1981237fba7e9748b3d43db95"}
Sep 15 07:51:01.830912 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.830791    2571 scope.go:117] "RemoveContainer" containerID="74ddd16112e4aa739bcdde7cf119e269ac64f2ba669a2f9489fb48efc5ee31b6"
Sep 15 07:51:01.832310 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.832288    2571 generic.go:356] "Generic (PLEG): container finished" podID="3e704c76-3c4d-4e0d-83b4-c0e9115f33c1" containerID="ccbc8d073f845e3dfd86da331719a0964903fbee4338bf95c37df81df291a1de" exitCode=0
Sep 15 07:51:01.832419 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.832336    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr"
Sep 15 07:51:01.832419 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.832357    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr" event={"ID":"3e704c76-3c4d-4e0d-83b4-c0e9115f33c1","Type":"ContainerDied","Data":"ccbc8d073f845e3dfd86da331719a0964903fbee4338bf95c37df81df291a1de"}
Sep 15 07:51:01.832419 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.832396    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr" event={"ID":"3e704c76-3c4d-4e0d-83b4-c0e9115f33c1","Type":"ContainerDied","Data":"c11ba94f96dd6918023db6fa5c99a5524f672556414d4128b17d79ec1627b248"}
Sep 15 07:51:01.839910 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.839888    2571 scope.go:117] "RemoveContainer" containerID="74ddd16112e4aa739bcdde7cf119e269ac64f2ba669a2f9489fb48efc5ee31b6"
Sep 15 07:51:01.840212 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:51:01.840186    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74ddd16112e4aa739bcdde7cf119e269ac64f2ba669a2f9489fb48efc5ee31b6\": container with ID starting with 74ddd16112e4aa739bcdde7cf119e269ac64f2ba669a2f9489fb48efc5ee31b6 not found: ID does not exist" containerID="74ddd16112e4aa739bcdde7cf119e269ac64f2ba669a2f9489fb48efc5ee31b6"
Sep 15 07:51:01.840341 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.840224    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74ddd16112e4aa739bcdde7cf119e269ac64f2ba669a2f9489fb48efc5ee31b6"} err="failed to get container status \"74ddd16112e4aa739bcdde7cf119e269ac64f2ba669a2f9489fb48efc5ee31b6\": rpc error: code = NotFound desc = could not find container \"74ddd16112e4aa739bcdde7cf119e269ac64f2ba669a2f9489fb48efc5ee31b6\": container with ID starting with 74ddd16112e4aa739bcdde7cf119e269ac64f2ba669a2f9489fb48efc5ee31b6 not found: ID does not exist"
Sep 15 07:51:01.840341 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.840280    2571 scope.go:117] "RemoveContainer" containerID="ccbc8d073f845e3dfd86da331719a0964903fbee4338bf95c37df81df291a1de"
Sep 15 07:51:01.849467 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.849447    2571 scope.go:117] "RemoveContainer" containerID="ccbc8d073f845e3dfd86da331719a0964903fbee4338bf95c37df81df291a1de"
Sep 15 07:51:01.849764 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:51:01.849743    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccbc8d073f845e3dfd86da331719a0964903fbee4338bf95c37df81df291a1de\": container with ID starting with ccbc8d073f845e3dfd86da331719a0964903fbee4338bf95c37df81df291a1de not found: ID does not exist" containerID="ccbc8d073f845e3dfd86da331719a0964903fbee4338bf95c37df81df291a1de"
Sep 15 07:51:01.849837 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.849771    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccbc8d073f845e3dfd86da331719a0964903fbee4338bf95c37df81df291a1de"} err="failed to get container status \"ccbc8d073f845e3dfd86da331719a0964903fbee4338bf95c37df81df291a1de\": rpc error: code = NotFound desc = could not find container \"ccbc8d073f845e3dfd86da331719a0964903fbee4338bf95c37df81df291a1de\": container with ID starting with ccbc8d073f845e3dfd86da331719a0964903fbee4338bf95c37df81df291a1de not found: ID does not exist"
Sep 15 07:51:01.865046 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.865019    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr"]
Sep 15 07:51:01.872084 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.872062    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f56165-7b58fd8f9d-68rdr"]
Sep 15 07:51:01.896493 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.896461    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26"]
Sep 15 07:51:01.900286 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.900261    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f56165-66fdc7d49c-wvq26"]
Sep 15 07:51:01.909922 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.909896    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e704c76-3c4d-4e0d-83b4-c0e9115f33c1" path="/var/lib/kubelet/pods/3e704c76-3c4d-4e0d-83b4-c0e9115f33c1/volumes"
Sep 15 07:51:01.910315 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:01.910300    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="827d14fe-b570-49ca-8600-778f1146b956" path="/var/lib/kubelet/pods/827d14fe-b570-49ca-8600-778f1146b956/volumes"
Sep 15 07:51:05.330523 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.330463    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk" podUID="411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0" containerName="main" containerID="cri-o://1d5485b158658939d03973841ba00a165ab53751d0be1dc8d36ed9720e5988c8" gracePeriod=45
Sep 15 07:51:05.351668 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.351630    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9" podUID="0f8b4cd6-013e-4a5d-ba1e-f39130dad902" containerName="main" containerID="cri-o://9d0dae549f0ef10fd888a37817d77e46a57c09f101440960962407f4265d39db" gracePeriod=45
Sep 15 07:51:05.590414 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.590348    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:51:05.628244 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.628217    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:51:05.639648 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.639619    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-home\") pod \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") "
Sep 15 07:51:05.639786 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.639687    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-model-cache\") pod \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") "
Sep 15 07:51:05.639786 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.639755    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-dshm\") pod \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") "
Sep 15 07:51:05.639898 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.639795    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvc6t\" (UniqueName: \"kubernetes.io/projected/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-kube-api-access-hvc6t\") pod \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") "
Sep 15 07:51:05.639898 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.639875    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-tmp-dir\") pod \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") "
Sep 15 07:51:05.640007 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.639909    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-home" (OuterVolumeSpecName: "home") pod "411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0" (UID: "411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:05.640007 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.639931    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0" (UID: "411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:05.640007 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.639922    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-tls-certs\") pod \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\" (UID: \"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0\") "
Sep 15 07:51:05.640431 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.640387    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0" (UID: "411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:05.640431 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.640408    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-home\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:05.640610 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.640445    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-model-cache\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:05.642414 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.642393    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-dshm" (OuterVolumeSpecName: "dshm") pod "411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0" (UID: "411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:05.642486 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.642432    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0" (UID: "411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:51:05.642520 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.642475    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-kube-api-access-hvc6t" (OuterVolumeSpecName: "kube-api-access-hvc6t") pod "411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0" (UID: "411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0"). InnerVolumeSpecName "kube-api-access-hvc6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:51:05.740791 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.740755    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-model-cache\") pod \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") "
Sep 15 07:51:05.740791 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.740796    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-home\") pod \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") "
Sep 15 07:51:05.741009 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.740839    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmkzl\" (UniqueName: \"kubernetes.io/projected/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-kube-api-access-vmkzl\") pod \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") "
Sep 15 07:51:05.741009 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.740937    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-dshm\") pod \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") "
Sep 15 07:51:05.741009 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.740982    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-tmp-dir\") pod \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") "
Sep 15 07:51:05.741127 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.741002    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f8b4cd6-013e-4a5d-ba1e-f39130dad902" (UID: "0f8b4cd6-013e-4a5d-ba1e-f39130dad902"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:05.741127 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.741039    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-tls-certs\") pod \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\" (UID: \"0f8b4cd6-013e-4a5d-ba1e-f39130dad902\") "
Sep 15 07:51:05.741127 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.741100    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-home" (OuterVolumeSpecName: "home") pod "0f8b4cd6-013e-4a5d-ba1e-f39130dad902" (UID: "0f8b4cd6-013e-4a5d-ba1e-f39130dad902"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:05.741293 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.741249    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f8b4cd6-013e-4a5d-ba1e-f39130dad902" (UID: "0f8b4cd6-013e-4a5d-ba1e-f39130dad902"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:05.741372 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.741353    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-dshm\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:05.741433 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.741380    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvc6t\" (UniqueName: \"kubernetes.io/projected/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-kube-api-access-hvc6t\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:05.741433 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.741391    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-tmp-dir\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:05.741433 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.741403    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0-tls-certs\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:05.741433 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.741413    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-tmp-dir\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:05.741433 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.741420    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-model-cache\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:05.741433 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.741427    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-home\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:05.743206 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.743187    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-dshm" (OuterVolumeSpecName: "dshm") pod "0f8b4cd6-013e-4a5d-ba1e-f39130dad902" (UID: "0f8b4cd6-013e-4a5d-ba1e-f39130dad902"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:05.743302 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.743198    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f8b4cd6-013e-4a5d-ba1e-f39130dad902" (UID: "0f8b4cd6-013e-4a5d-ba1e-f39130dad902"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:51:05.743302 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.743212    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-kube-api-access-vmkzl" (OuterVolumeSpecName: "kube-api-access-vmkzl") pod "0f8b4cd6-013e-4a5d-ba1e-f39130dad902" (UID: "0f8b4cd6-013e-4a5d-ba1e-f39130dad902"). InnerVolumeSpecName "kube-api-access-vmkzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:51:05.842847 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.842761    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-dshm\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:05.842847 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.842789    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-tls-certs\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:05.842847 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.842801    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmkzl\" (UniqueName: \"kubernetes.io/projected/0f8b4cd6-013e-4a5d-ba1e-f39130dad902-kube-api-access-vmkzl\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:05.859231 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.859194    2571 generic.go:356] "Generic (PLEG): container finished" podID="411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0" containerID="1d5485b158658939d03973841ba00a165ab53751d0be1dc8d36ed9720e5988c8" exitCode=0
Sep 15 07:51:05.859430 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.859237    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"
Sep 15 07:51:05.859430 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.859278    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk" event={"ID":"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0","Type":"ContainerDied","Data":"1d5485b158658939d03973841ba00a165ab53751d0be1dc8d36ed9720e5988c8"}
Sep 15 07:51:05.859430 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.859324    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk" event={"ID":"411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0","Type":"ContainerDied","Data":"ef6a665c395cb9e9a0dc3ad0a9bb7f48894b21f0af42b420fa28b8c1c806e698"}
Sep 15 07:51:05.859430 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.859343    2571 scope.go:117] "RemoveContainer" containerID="1d5485b158658939d03973841ba00a165ab53751d0be1dc8d36ed9720e5988c8"
Sep 15 07:51:05.860923 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.860892    2571 generic.go:356] "Generic (PLEG): container finished" podID="0f8b4cd6-013e-4a5d-ba1e-f39130dad902" containerID="9d0dae549f0ef10fd888a37817d77e46a57c09f101440960962407f4265d39db" exitCode=0
Sep 15 07:51:05.861027 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.860927    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"
Sep 15 07:51:05.861027 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.860930    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9" event={"ID":"0f8b4cd6-013e-4a5d-ba1e-f39130dad902","Type":"ContainerDied","Data":"9d0dae549f0ef10fd888a37817d77e46a57c09f101440960962407f4265d39db"}
Sep 15 07:51:05.861027 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.860959    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9" event={"ID":"0f8b4cd6-013e-4a5d-ba1e-f39130dad902","Type":"ContainerDied","Data":"b3ac1f0aa4c2197b2552fd390c93e318d51fe5a32286c423750639e66c43c9a1"}
Sep 15 07:51:05.868929 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.868911    2571 scope.go:117] "RemoveContainer" containerID="1d5485b158658939d03973841ba00a165ab53751d0be1dc8d36ed9720e5988c8"
Sep 15 07:51:05.869194 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:51:05.869176    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d5485b158658939d03973841ba00a165ab53751d0be1dc8d36ed9720e5988c8\": container with ID starting with 1d5485b158658939d03973841ba00a165ab53751d0be1dc8d36ed9720e5988c8 not found: ID does not exist" containerID="1d5485b158658939d03973841ba00a165ab53751d0be1dc8d36ed9720e5988c8"
Sep 15 07:51:05.869246 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.869204    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d5485b158658939d03973841ba00a165ab53751d0be1dc8d36ed9720e5988c8"} err="failed to get container status \"1d5485b158658939d03973841ba00a165ab53751d0be1dc8d36ed9720e5988c8\": rpc error: code = NotFound desc = could not find container \"1d5485b158658939d03973841ba00a165ab53751d0be1dc8d36ed9720e5988c8\": container with ID starting with 1d5485b158658939d03973841ba00a165ab53751d0be1dc8d36ed9720e5988c8 not found: ID does not exist"
Sep 15 07:51:05.869246 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.869222    2571 scope.go:117] "RemoveContainer" containerID="9d0dae549f0ef10fd888a37817d77e46a57c09f101440960962407f4265d39db"
Sep 15 07:51:05.877730 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.877716    2571 scope.go:117] "RemoveContainer" containerID="9d0dae549f0ef10fd888a37817d77e46a57c09f101440960962407f4265d39db"
Sep 15 07:51:05.877987 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:51:05.877971    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d0dae549f0ef10fd888a37817d77e46a57c09f101440960962407f4265d39db\": container with ID starting with 9d0dae549f0ef10fd888a37817d77e46a57c09f101440960962407f4265d39db not found: ID does not exist" containerID="9d0dae549f0ef10fd888a37817d77e46a57c09f101440960962407f4265d39db"
Sep 15 07:51:05.878038 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.877993    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d0dae549f0ef10fd888a37817d77e46a57c09f101440960962407f4265d39db"} err="failed to get container status \"9d0dae549f0ef10fd888a37817d77e46a57c09f101440960962407f4265d39db\": rpc error: code = NotFound desc = could not find container \"9d0dae549f0ef10fd888a37817d77e46a57c09f101440960962407f4265d39db\": container with ID starting with 9d0dae549f0ef10fd888a37817d77e46a57c09f101440960962407f4265d39db not found: ID does not exist"
Sep 15 07:51:05.883586 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.883560    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"]
Sep 15 07:51:05.888603 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.888582    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/simulator-w4-f56165-kserve-7f4d9f684b-2rpt9"]
Sep 15 07:51:05.898577 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.898554    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"]
Sep 15 07:51:05.900397 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.900378    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/distinct-w4-f56165-kserve-7d6bbd46db-fr4rk"]
Sep 15 07:51:05.913966 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.913935    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f8b4cd6-013e-4a5d-ba1e-f39130dad902" path="/var/lib/kubelet/pods/0f8b4cd6-013e-4a5d-ba1e-f39130dad902/volumes"
Sep 15 07:51:05.914637 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:05.914611    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0" path="/var/lib/kubelet/pods/411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0/volumes"
Sep 15 07:51:16.188225 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:16.188147    2571 scope.go:117] "RemoveContainer" containerID="cd64ba6ac169ac0276996430de60cd3967705a72dff2b0de2c4651935296e934"
Sep 15 07:51:16.196616 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:16.196598    2571 scope.go:117] "RemoveContainer" containerID="610c016083d20f4bd4693c5c0ef6c882e5d2dfcea64c802f213afd40b8514aba"
Sep 15 07:51:20.797589 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:20.797549    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"]
Sep 15 07:51:22.395778 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:22.395745    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"]
Sep 15 07:51:35.817512 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:35.817461    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4" podUID="ea796e0e-601f-4aa1-aa8e-e01673fbe40f" containerName="main" containerID="cri-o://386dbd0e5945cf6c7e0c8944e1803aa302b0ceb2c601ee2ef7b7d3a25a780880" gracePeriod=45
Sep 15 07:51:36.023122 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:36.023090    2571 generic.go:356] "Generic (PLEG): container finished" podID="ea796e0e-601f-4aa1-aa8e-e01673fbe40f" containerID="386dbd0e5945cf6c7e0c8944e1803aa302b0ceb2c601ee2ef7b7d3a25a780880" exitCode=0
Sep 15 07:51:36.023301 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:36.023160    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4" event={"ID":"ea796e0e-601f-4aa1-aa8e-e01673fbe40f","Type":"ContainerDied","Data":"386dbd0e5945cf6c7e0c8944e1803aa302b0ceb2c601ee2ef7b7d3a25a780880"}
Sep 15 07:51:36.057915 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:36.057892    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:51:36.145793 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:36.145709    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-model-cache\") pod \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") "
Sep 15 07:51:36.145793 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:36.145767    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-tmp-dir\") pod \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") "
Sep 15 07:51:36.145793 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:36.145793    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-tls-certs\") pod \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") "
Sep 15 07:51:36.146074 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:36.145822    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksrht\" (UniqueName: \"kubernetes.io/projected/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-kube-api-access-ksrht\") pod \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") "
Sep 15 07:51:36.146074 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:36.145854    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-dshm\") pod \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") "
Sep 15 07:51:36.146074 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:36.145885    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-home\") pod \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\" (UID: \"ea796e0e-601f-4aa1-aa8e-e01673fbe40f\") "
Sep 15 07:51:36.146074 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:36.146042    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea796e0e-601f-4aa1-aa8e-e01673fbe40f" (UID: "ea796e0e-601f-4aa1-aa8e-e01673fbe40f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:36.146074 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:36.146053    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea796e0e-601f-4aa1-aa8e-e01673fbe40f" (UID: "ea796e0e-601f-4aa1-aa8e-e01673fbe40f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:36.146363 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:36.146246    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-model-cache\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:36.146363 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:36.146295    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-tmp-dir\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:36.146363 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:36.146307    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-home" (OuterVolumeSpecName: "home") pod "ea796e0e-601f-4aa1-aa8e-e01673fbe40f" (UID: "ea796e0e-601f-4aa1-aa8e-e01673fbe40f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:36.148136 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:36.148107    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-dshm" (OuterVolumeSpecName: "dshm") pod "ea796e0e-601f-4aa1-aa8e-e01673fbe40f" (UID: "ea796e0e-601f-4aa1-aa8e-e01673fbe40f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:36.148265 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:36.148145    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea796e0e-601f-4aa1-aa8e-e01673fbe40f" (UID: "ea796e0e-601f-4aa1-aa8e-e01673fbe40f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:51:36.148265 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:36.148175    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-kube-api-access-ksrht" (OuterVolumeSpecName: "kube-api-access-ksrht") pod "ea796e0e-601f-4aa1-aa8e-e01673fbe40f" (UID: "ea796e0e-601f-4aa1-aa8e-e01673fbe40f"). InnerVolumeSpecName "kube-api-access-ksrht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:51:36.247325 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:36.247289    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-tls-certs\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:36.247325 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:36.247320    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksrht\" (UniqueName: \"kubernetes.io/projected/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-kube-api-access-ksrht\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:36.247325 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:36.247330    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-dshm\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:36.247555 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:36.247339    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea796e0e-601f-4aa1-aa8e-e01673fbe40f-home\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:37.029455 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.029423    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4" event={"ID":"ea796e0e-601f-4aa1-aa8e-e01673fbe40f","Type":"ContainerDied","Data":"f6810cc6229a4f1dd48c3252c8b725f501285683e0b069a21f974b70ba4a8e3d"}
Sep 15 07:51:37.029455 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.029454    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"
Sep 15 07:51:37.029956 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.029475    2571 scope.go:117] "RemoveContainer" containerID="386dbd0e5945cf6c7e0c8944e1803aa302b0ceb2c601ee2ef7b7d3a25a780880"
Sep 15 07:51:37.051229 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.051201    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"]
Sep 15 07:51:37.055931 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.055906    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-04464b/auth-test-model-04464b-kserve-6ddb8bb859-m7gk4"]
Sep 15 07:51:37.416918 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.416834    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb" podUID="bf5d46da-426d-4dca-bc70-982b481cb5d8" containerName="main" containerID="cri-o://e0ffa50405e1ec3a91b4c6dd38b7d4e5f415d736661f67db223882b363b3c634" gracePeriod=45
Sep 15 07:51:37.673610 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.673546    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:51:37.759434 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.759398    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv79n\" (UniqueName: \"kubernetes.io/projected/bf5d46da-426d-4dca-bc70-982b481cb5d8-kube-api-access-mv79n\") pod \"bf5d46da-426d-4dca-bc70-982b481cb5d8\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") "
Sep 15 07:51:37.759611 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.759474    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-model-cache\") pod \"bf5d46da-426d-4dca-bc70-982b481cb5d8\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") "
Sep 15 07:51:37.759611 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.759527    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-dshm\") pod \"bf5d46da-426d-4dca-bc70-982b481cb5d8\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") "
Sep 15 07:51:37.759611 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.759562    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5d46da-426d-4dca-bc70-982b481cb5d8-tls-certs\") pod \"bf5d46da-426d-4dca-bc70-982b481cb5d8\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") "
Sep 15 07:51:37.759611 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.759586    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-home\") pod \"bf5d46da-426d-4dca-bc70-982b481cb5d8\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") "
Sep 15 07:51:37.759832 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.759618    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-tmp-dir\") pod \"bf5d46da-426d-4dca-bc70-982b481cb5d8\" (UID: \"bf5d46da-426d-4dca-bc70-982b481cb5d8\") "
Sep 15 07:51:37.759832 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.759793    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf5d46da-426d-4dca-bc70-982b481cb5d8" (UID: "bf5d46da-426d-4dca-bc70-982b481cb5d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:37.759941 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.759847    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-home" (OuterVolumeSpecName: "home") pod "bf5d46da-426d-4dca-bc70-982b481cb5d8" (UID: "bf5d46da-426d-4dca-bc70-982b481cb5d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:37.760010 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.759983    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf5d46da-426d-4dca-bc70-982b481cb5d8" (UID: "bf5d46da-426d-4dca-bc70-982b481cb5d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:37.760060 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.760016    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-model-cache\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:37.760060 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.760029    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-home\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:37.761769 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.761739    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5d46da-426d-4dca-bc70-982b481cb5d8-kube-api-access-mv79n" (OuterVolumeSpecName: "kube-api-access-mv79n") pod "bf5d46da-426d-4dca-bc70-982b481cb5d8" (UID: "bf5d46da-426d-4dca-bc70-982b481cb5d8"). InnerVolumeSpecName "kube-api-access-mv79n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:51:37.761891 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.761742    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf5d46da-426d-4dca-bc70-982b481cb5d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf5d46da-426d-4dca-bc70-982b481cb5d8" (UID: "bf5d46da-426d-4dca-bc70-982b481cb5d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:51:37.761891 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.761745    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-dshm" (OuterVolumeSpecName: "dshm") pod "bf5d46da-426d-4dca-bc70-982b481cb5d8" (UID: "bf5d46da-426d-4dca-bc70-982b481cb5d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:37.860960 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.860924    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-dshm\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:37.860960 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.860952    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5d46da-426d-4dca-bc70-982b481cb5d8-tls-certs\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:37.860960 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.860962    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5d46da-426d-4dca-bc70-982b481cb5d8-tmp-dir\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:37.861185 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.860971    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv79n\" (UniqueName: \"kubernetes.io/projected/bf5d46da-426d-4dca-bc70-982b481cb5d8-kube-api-access-mv79n\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:51:37.909934 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:37.909899    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea796e0e-601f-4aa1-aa8e-e01673fbe40f" path="/var/lib/kubelet/pods/ea796e0e-601f-4aa1-aa8e-e01673fbe40f/volumes"
Sep 15 07:51:38.036810 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:38.036774    2571 generic.go:356] "Generic (PLEG): container finished" podID="bf5d46da-426d-4dca-bc70-982b481cb5d8" containerID="e0ffa50405e1ec3a91b4c6dd38b7d4e5f415d736661f67db223882b363b3c634" exitCode=0
Sep 15 07:51:38.037306 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:38.036829    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"
Sep 15 07:51:38.037306 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:38.036845    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb" event={"ID":"bf5d46da-426d-4dca-bc70-982b481cb5d8","Type":"ContainerDied","Data":"e0ffa50405e1ec3a91b4c6dd38b7d4e5f415d736661f67db223882b363b3c634"}
Sep 15 07:51:38.037306 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:38.036884    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb" event={"ID":"bf5d46da-426d-4dca-bc70-982b481cb5d8","Type":"ContainerDied","Data":"80ad3b02acdf0e5f3d42864489619632b5119c8050b83aa3589d6e20f7399a79"}
Sep 15 07:51:38.037306 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:38.036910    2571 scope.go:117] "RemoveContainer" containerID="e0ffa50405e1ec3a91b4c6dd38b7d4e5f415d736661f67db223882b363b3c634"
Sep 15 07:51:38.046164 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:38.046138    2571 scope.go:117] "RemoveContainer" containerID="e0ffa50405e1ec3a91b4c6dd38b7d4e5f415d736661f67db223882b363b3c634"
Sep 15 07:51:38.046474 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:51:38.046450    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0ffa50405e1ec3a91b4c6dd38b7d4e5f415d736661f67db223882b363b3c634\": container with ID starting with e0ffa50405e1ec3a91b4c6dd38b7d4e5f415d736661f67db223882b363b3c634 not found: ID does not exist" containerID="e0ffa50405e1ec3a91b4c6dd38b7d4e5f415d736661f67db223882b363b3c634"
Sep 15 07:51:38.046539 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:38.046486    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ffa50405e1ec3a91b4c6dd38b7d4e5f415d736661f67db223882b363b3c634"} err="failed to get container status \"e0ffa50405e1ec3a91b4c6dd38b7d4e5f415d736661f67db223882b363b3c634\": rpc error: code = NotFound desc = could not find container \"e0ffa50405e1ec3a91b4c6dd38b7d4e5f415d736661f67db223882b363b3c634\": container with ID starting with e0ffa50405e1ec3a91b4c6dd38b7d4e5f415d736661f67db223882b363b3c634 not found: ID does not exist"
Sep 15 07:51:38.054958 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:38.054934    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"]
Sep 15 07:51:38.060805 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:38.060783    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e3a482/auth-test-model-e3a482-kserve-769cb764f9-qrhjb"]
Sep 15 07:51:39.910571 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:51:39.910536    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf5d46da-426d-4dca-bc70-982b481cb5d8" path="/var/lib/kubelet/pods/bf5d46da-426d-4dca-bc70-982b481cb5d8/volumes"
Sep 15 07:52:16.244485 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:16.244452    2571 scope.go:117] "RemoveContainer" containerID="769b98712ac3d59f124aeef251d478f15b8f1933ec3cb7f54bd43fa07c631acc"
Sep 15 07:52:22.695208 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695175    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7660-d29195"]
Sep 15 07:52:22.695661 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695600    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f8b4cd6-013e-4a5d-ba1e-f39130dad902" containerName="main"
Sep 15 07:52:22.695661 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695616    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f8b4cd6-013e-4a5d-ba1e-f39130dad902" containerName="main"
Sep 15 07:52:22.695661 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695630    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c672cec7-2811-4072-961f-9dd921e5aef5" containerName="payload-processing"
Sep 15 07:52:22.695661 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695636    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c672cec7-2811-4072-961f-9dd921e5aef5" containerName="payload-processing"
Sep 15 07:52:22.695661 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695645    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0" containerName="main"
Sep 15 07:52:22.695661 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695650    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0" containerName="main"
Sep 15 07:52:22.695661 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695657    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf5d46da-426d-4dca-bc70-982b481cb5d8" containerName="main"
Sep 15 07:52:22.695661 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695662    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf5d46da-426d-4dca-bc70-982b481cb5d8" containerName="main"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695673    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a09614a-200b-4d0d-bc0f-59b1313f3c62" containerName="payload-pre-processing"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695678    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a09614a-200b-4d0d-bc0f-59b1313f3c62" containerName="payload-pre-processing"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695695    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b53d9786-e345-4c0e-95c1-3980704dbc36" containerName="revoke-keys"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695704    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b53d9786-e345-4c0e-95c1-3980704dbc36" containerName="revoke-keys"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695712    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="827d14fe-b570-49ca-8600-778f1146b956" containerName="payload-processing"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695717    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="827d14fe-b570-49ca-8600-778f1146b956" containerName="payload-processing"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695724    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea796e0e-601f-4aa1-aa8e-e01673fbe40f" containerName="main"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695729    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea796e0e-601f-4aa1-aa8e-e01673fbe40f" containerName="main"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695742    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b2502c9-e9d7-4ecf-b431-cd82e5a793fa" containerName="payload-processing"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695747    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b2502c9-e9d7-4ecf-b431-cd82e5a793fa" containerName="payload-processing"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695754    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e704c76-3c4d-4e0d-83b4-c0e9115f33c1" containerName="payload-pre-processing"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695759    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e704c76-3c4d-4e0d-83b4-c0e9115f33c1" containerName="payload-pre-processing"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695767    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f1cb03-c1e7-4820-934d-62efeb27d9ea" containerName="payload-pre-processing"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695775    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f1cb03-c1e7-4820-934d-62efeb27d9ea" containerName="payload-pre-processing"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695834    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f8b4cd6-013e-4a5d-ba1e-f39130dad902" containerName="main"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695842    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b53d9786-e345-4c0e-95c1-3980704dbc36" containerName="revoke-keys"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695849    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c672cec7-2811-4072-961f-9dd921e5aef5" containerName="payload-processing"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695857    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e704c76-3c4d-4e0d-83b4-c0e9115f33c1" containerName="payload-pre-processing"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695865    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f1cb03-c1e7-4820-934d-62efeb27d9ea" containerName="payload-pre-processing"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695872    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf5d46da-426d-4dca-bc70-982b481cb5d8" containerName="main"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695877    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a09614a-200b-4d0d-bc0f-59b1313f3c62" containerName="payload-pre-processing"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695883    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b2502c9-e9d7-4ecf-b431-cd82e5a793fa" containerName="payload-processing"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695890    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea796e0e-601f-4aa1-aa8e-e01673fbe40f" containerName="main"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695896    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="827d14fe-b570-49ca-8600-778f1146b956" containerName="payload-processing"
Sep 15 07:52:22.695933 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.695901    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="411641e0-2bbf-4ddf-bbf4-fce8b8ed41f0" containerName="main"
Sep 15 07:52:22.698912 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.698896    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-d29195"
Sep 15 07:52:22.701235 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.701210    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-x8zjf\""
Sep 15 07:52:22.706741 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.706715    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7660-d29195"]
Sep 15 07:52:22.778001 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.777967    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tv74\" (UniqueName: \"kubernetes.io/projected/43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99-kube-api-access-8tv74\") pod \"test-curl-7660-d29195\" (UID: \"43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99\") " pod="opendatahub/test-curl-7660-d29195"
Sep 15 07:52:22.878536 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.878503    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tv74\" (UniqueName: \"kubernetes.io/projected/43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99-kube-api-access-8tv74\") pod \"test-curl-7660-d29195\" (UID: \"43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99\") " pod="opendatahub/test-curl-7660-d29195"
Sep 15 07:52:22.888211 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:22.888180    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tv74\" (UniqueName: \"kubernetes.io/projected/43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99-kube-api-access-8tv74\") pod \"test-curl-7660-d29195\" (UID: \"43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99\") " pod="opendatahub/test-curl-7660-d29195"
Sep 15 07:52:23.017882 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:23.017840    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-d29195"
Sep 15 07:52:23.288596 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:23.288557    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7660-d29195"]
Sep 15 07:52:23.290903 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:52:23.290875    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43aac3cc_d0ad_4b44_a6aa_0fc10ab40f99.slice/crio-7556bc26984ddc97a093e545355f17fa1428e3985ac3a49907596530dddfeaf6 WatchSource:0}: Error finding container 7556bc26984ddc97a093e545355f17fa1428e3985ac3a49907596530dddfeaf6: Status 404 returned error can't find the container with id 7556bc26984ddc97a093e545355f17fa1428e3985ac3a49907596530dddfeaf6
Sep 15 07:52:23.292694 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:23.292677    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 07:52:24.279093 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:24.279044    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7660-d29195" event={"ID":"43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99","Type":"ContainerStarted","Data":"7556bc26984ddc97a093e545355f17fa1428e3985ac3a49907596530dddfeaf6"}
Sep 15 07:52:26.293242 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:26.293206    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7660-d29195" event={"ID":"43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99","Type":"ContainerStarted","Data":"faddc32216ab3822c98785c31e5bd3e99c67992004be6ea4ccff95f5245cd1fe"}
Sep 15 07:52:26.308377 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:26.308327    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7660-d29195" podStartSLOduration=2.153636171 podStartE2EDuration="4.308311259s" totalImagesPullingTime="2.154675088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:52:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:52:26.30737133 +0000 UTC m=+2050.853941428" watchObservedRunningTime="2026-09-15 07:52:26.308311259 +0000 UTC m=+2050.854881358"
Sep 15 07:52:26.901992 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:26.901956    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7660-d29195"]
Sep 15 07:52:26.907675 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:26.907646    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7660-d29195"]
Sep 15 07:52:27.299015 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:52:27.298965    2571 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7660-d29195" container="test-curl-7660-d29195"
Sep 15 07:52:28.303695 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:28.303653    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7660-d29195" podUID="43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99" containerName="test-curl-7660-d29195" containerID="cri-o://faddc32216ab3822c98785c31e5bd3e99c67992004be6ea4ccff95f5245cd1fe" gracePeriod=2
Sep 15 07:52:30.474127 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:30.474100    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-d29195"
Sep 15 07:52:30.553574 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:30.553543    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tv74\" (UniqueName: \"kubernetes.io/projected/43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99-kube-api-access-8tv74\") pod \"43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99\" (UID: \"43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99\") "
Sep 15 07:52:30.555724 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:30.555699    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99-kube-api-access-8tv74" (OuterVolumeSpecName: "kube-api-access-8tv74") pod "43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99" (UID: "43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99"). InnerVolumeSpecName "kube-api-access-8tv74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:52:30.654283 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:30.654201    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tv74\" (UniqueName: \"kubernetes.io/projected/43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99-kube-api-access-8tv74\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:52:31.320793 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:31.320756    2571 generic.go:356] "Generic (PLEG): container finished" podID="43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99" containerID="faddc32216ab3822c98785c31e5bd3e99c67992004be6ea4ccff95f5245cd1fe" exitCode=137
Sep 15 07:52:31.320979 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:31.320810    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-d29195"
Sep 15 07:52:31.320979 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:31.320861    2571 scope.go:117] "RemoveContainer" containerID="faddc32216ab3822c98785c31e5bd3e99c67992004be6ea4ccff95f5245cd1fe"
Sep 15 07:52:31.330170 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:31.330150    2571 scope.go:117] "RemoveContainer" containerID="faddc32216ab3822c98785c31e5bd3e99c67992004be6ea4ccff95f5245cd1fe"
Sep 15 07:52:31.330474 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:52:31.330454    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faddc32216ab3822c98785c31e5bd3e99c67992004be6ea4ccff95f5245cd1fe\": container with ID starting with faddc32216ab3822c98785c31e5bd3e99c67992004be6ea4ccff95f5245cd1fe not found: ID does not exist" containerID="faddc32216ab3822c98785c31e5bd3e99c67992004be6ea4ccff95f5245cd1fe"
Sep 15 07:52:31.330531 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:31.330483    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faddc32216ab3822c98785c31e5bd3e99c67992004be6ea4ccff95f5245cd1fe"} err="failed to get container status \"faddc32216ab3822c98785c31e5bd3e99c67992004be6ea4ccff95f5245cd1fe\": rpc error: code = NotFound desc = could not find container \"faddc32216ab3822c98785c31e5bd3e99c67992004be6ea4ccff95f5245cd1fe\": container with ID starting with faddc32216ab3822c98785c31e5bd3e99c67992004be6ea4ccff95f5245cd1fe not found: ID does not exist"
Sep 15 07:52:31.331077 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:52:31.331054    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7660-d29195\" is forbidden: User \"system:node:ip-10-0-129-76.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-76.ec2.internal' and this object" podUID="43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99" pod="opendatahub/test-curl-7660-d29195"
Sep 15 07:52:31.908873 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:31.908839    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99" path="/var/lib/kubelet/pods/43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99/volumes"
Sep 15 07:52:43.212455 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.212415    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"]
Sep 15 07:52:43.212926 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.212825    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99" containerName="test-curl-7660-d29195"
Sep 15 07:52:43.212926 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.212837    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99" containerName="test-curl-7660-d29195"
Sep 15 07:52:43.212926 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.212906    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="43aac3cc-d0ad-4b44-a6aa-0fc10ab40f99" containerName="test-curl-7660-d29195"
Sep 15 07:52:43.217618 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.217597    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:43.220418 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.220396    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-04464b\"/\"openshift-service-ca.crt\""
Sep 15 07:52:43.221552 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.221531    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-04464b\"/\"default-dockercfg-79sv4\""
Sep 15 07:52:43.221649 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.221539    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-04464b\"/\"kube-root-ca.crt\""
Sep 15 07:52:43.221649 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.221588    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-04464b\"/\"rate-test-model-04464b-kserve-self-signed-certs\""
Sep 15 07:52:43.226064 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.226039    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"]
Sep 15 07:52:43.379171 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.379131    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf281d2-ecce-4ad9-95b5-383684a662f6-tls-certs\") pod \"rate-test-model-04464b-kserve-9bb57c999-29xsp\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") " pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:43.379402 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.379180    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4qz6\" (UniqueName: \"kubernetes.io/projected/ebf281d2-ecce-4ad9-95b5-383684a662f6-kube-api-access-k4qz6\") pod \"rate-test-model-04464b-kserve-9bb57c999-29xsp\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") " pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:43.379402 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.379235    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-model-cache\") pod \"rate-test-model-04464b-kserve-9bb57c999-29xsp\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") " pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:43.379402 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.379287    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-home\") pod \"rate-test-model-04464b-kserve-9bb57c999-29xsp\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") " pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:43.379402 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.379324    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-dshm\") pod \"rate-test-model-04464b-kserve-9bb57c999-29xsp\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") " pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:43.379402 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.379343    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-tmp-dir\") pod \"rate-test-model-04464b-kserve-9bb57c999-29xsp\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") " pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:43.480441 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.480362    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf281d2-ecce-4ad9-95b5-383684a662f6-tls-certs\") pod \"rate-test-model-04464b-kserve-9bb57c999-29xsp\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") " pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:43.480441 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.480406    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4qz6\" (UniqueName: \"kubernetes.io/projected/ebf281d2-ecce-4ad9-95b5-383684a662f6-kube-api-access-k4qz6\") pod \"rate-test-model-04464b-kserve-9bb57c999-29xsp\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") " pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:43.480627 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.480456    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-model-cache\") pod \"rate-test-model-04464b-kserve-9bb57c999-29xsp\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") " pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:43.480627 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.480485    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-home\") pod \"rate-test-model-04464b-kserve-9bb57c999-29xsp\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") " pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:43.480627 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.480614    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-dshm\") pod \"rate-test-model-04464b-kserve-9bb57c999-29xsp\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") " pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:43.480737 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.480657    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-tmp-dir\") pod \"rate-test-model-04464b-kserve-9bb57c999-29xsp\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") " pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:43.480904 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.480885    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-home\") pod \"rate-test-model-04464b-kserve-9bb57c999-29xsp\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") " pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:43.480998 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.480942    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-model-cache\") pod \"rate-test-model-04464b-kserve-9bb57c999-29xsp\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") " pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:43.481077 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.481055    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-tmp-dir\") pod \"rate-test-model-04464b-kserve-9bb57c999-29xsp\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") " pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:43.482764 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.482743    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-dshm\") pod \"rate-test-model-04464b-kserve-9bb57c999-29xsp\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") " pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:43.482947 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.482932    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf281d2-ecce-4ad9-95b5-383684a662f6-tls-certs\") pod \"rate-test-model-04464b-kserve-9bb57c999-29xsp\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") " pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:43.487778 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.487755    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4qz6\" (UniqueName: \"kubernetes.io/projected/ebf281d2-ecce-4ad9-95b5-383684a662f6-kube-api-access-k4qz6\") pod \"rate-test-model-04464b-kserve-9bb57c999-29xsp\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") " pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:43.528748 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.528720    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:43.798996 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:43.798971    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"]
Sep 15 07:52:43.800914 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:52:43.800885    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebf281d2_ecce_4ad9_95b5_383684a662f6.slice/crio-20899c995af88f0680531fddde3c79dafe80cc71d18cb5e08a81d763ea11d848 WatchSource:0}: Error finding container 20899c995af88f0680531fddde3c79dafe80cc71d18cb5e08a81d763ea11d848: Status 404 returned error can't find the container with id 20899c995af88f0680531fddde3c79dafe80cc71d18cb5e08a81d763ea11d848
Sep 15 07:52:44.389601 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:44.389557    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp" event={"ID":"ebf281d2-ecce-4ad9-95b5-383684a662f6","Type":"ContainerStarted","Data":"c923af19ec8f6c8e0f100b61d1db8c53d5231cbbdf511ebbb3a82cf3c91fa5ab"}
Sep 15 07:52:44.389991 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:44.389611    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp" event={"ID":"ebf281d2-ecce-4ad9-95b5-383684a662f6","Type":"ContainerStarted","Data":"20899c995af88f0680531fddde3c79dafe80cc71d18cb5e08a81d763ea11d848"}
Sep 15 07:52:44.407307 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:44.407235    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp" podStartSLOduration=1.4072153680000001 podStartE2EDuration="1.407215368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:52:44.406242053 +0000 UTC m=+2068.952812173" watchObservedRunningTime="2026-09-15 07:52:44.407215368 +0000 UTC m=+2068.953785468"
Sep 15 07:52:53.529339 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:53.529294    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:53.529339 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:53.529347    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:53.541926 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:53.541898    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:52:54.451554 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:52:54.451516    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:54:01.915153 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:01.915120    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"]
Sep 15 07:54:09.044023 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.043940    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9"]
Sep 15 07:54:09.047763 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.047741    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9"
Sep 15 07:54:09.050066 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.050041    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b179ed\""
Sep 15 07:54:09.050181 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.050078    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b179ed-dockercfg-7kgtw\""
Sep 15 07:54:09.062407 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.062377    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9"]
Sep 15 07:54:09.070905 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.070875    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb"]
Sep 15 07:54:09.074446 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.074429    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb"
Sep 15 07:54:09.083381 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.083353    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb"]
Sep 15 07:54:09.117918 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.117878    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dqtq\" (UniqueName: \"kubernetes.io/projected/ab4409e7-24c0-465b-b4f0-97481a9d5391-kube-api-access-5dqtq\") pod \"payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb\" (UID: \"ab4409e7-24c0-465b-b4f0-97481a9d5391\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb"
Sep 15 07:54:09.118086 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.118005    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8000dd9-435e-4444-9a3c-5cfd153b74c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9\" (UID: \"d8000dd9-435e-4444-9a3c-5cfd153b74c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9"
Sep 15 07:54:09.118086 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.118040    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4409e7-24c0-465b-b4f0-97481a9d5391-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb\" (UID: \"ab4409e7-24c0-465b-b4f0-97481a9d5391\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb"
Sep 15 07:54:09.118178 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.118093    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vvkx\" (UniqueName: \"kubernetes.io/projected/d8000dd9-435e-4444-9a3c-5cfd153b74c6-kube-api-access-2vvkx\") pod \"payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9\" (UID: \"d8000dd9-435e-4444-9a3c-5cfd153b74c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9"
Sep 15 07:54:09.218701 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.218664    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vvkx\" (UniqueName: \"kubernetes.io/projected/d8000dd9-435e-4444-9a3c-5cfd153b74c6-kube-api-access-2vvkx\") pod \"payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9\" (UID: \"d8000dd9-435e-4444-9a3c-5cfd153b74c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9"
Sep 15 07:54:09.218876 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.218730    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dqtq\" (UniqueName: \"kubernetes.io/projected/ab4409e7-24c0-465b-b4f0-97481a9d5391-kube-api-access-5dqtq\") pod \"payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb\" (UID: \"ab4409e7-24c0-465b-b4f0-97481a9d5391\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb"
Sep 15 07:54:09.218876 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.218794    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8000dd9-435e-4444-9a3c-5cfd153b74c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9\" (UID: \"d8000dd9-435e-4444-9a3c-5cfd153b74c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9"
Sep 15 07:54:09.218876 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.218837    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4409e7-24c0-465b-b4f0-97481a9d5391-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb\" (UID: \"ab4409e7-24c0-465b-b4f0-97481a9d5391\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb"
Sep 15 07:54:09.219543 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.219524    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4409e7-24c0-465b-b4f0-97481a9d5391-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb\" (UID: \"ab4409e7-24c0-465b-b4f0-97481a9d5391\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb"
Sep 15 07:54:09.219715 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.219690    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8000dd9-435e-4444-9a3c-5cfd153b74c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9\" (UID: \"d8000dd9-435e-4444-9a3c-5cfd153b74c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9"
Sep 15 07:54:09.226629 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.226600    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vvkx\" (UniqueName: \"kubernetes.io/projected/d8000dd9-435e-4444-9a3c-5cfd153b74c6-kube-api-access-2vvkx\") pod \"payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9\" (UID: \"d8000dd9-435e-4444-9a3c-5cfd153b74c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9"
Sep 15 07:54:09.226740 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.226647    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dqtq\" (UniqueName: \"kubernetes.io/projected/ab4409e7-24c0-465b-b4f0-97481a9d5391-kube-api-access-5dqtq\") pod \"payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb\" (UID: \"ab4409e7-24c0-465b-b4f0-97481a9d5391\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb"
Sep 15 07:54:09.357934 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.357841    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9"
Sep 15 07:54:09.387523 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.387488    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb"
Sep 15 07:54:09.635233 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.635208    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9"]
Sep 15 07:54:09.637327 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:54:09.637300    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8000dd9_435e_4444_9a3c_5cfd153b74c6.slice/crio-10a306f4d95a4a671131d46083a4a752c77aa5a7c4340827b07e473a81927dfc WatchSource:0}: Error finding container 10a306f4d95a4a671131d46083a4a752c77aa5a7c4340827b07e473a81927dfc: Status 404 returned error can't find the container with id 10a306f4d95a4a671131d46083a4a752c77aa5a7c4340827b07e473a81927dfc
Sep 15 07:54:09.668649 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.668620    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb"]
Sep 15 07:54:09.670319 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:54:09.670287    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4409e7_24c0_465b_b4f0_97481a9d5391.slice/crio-ad5aa22075be573c86f4011338a58ed5f5fcb34badc7f887525ba72805f2e480 WatchSource:0}: Error finding container ad5aa22075be573c86f4011338a58ed5f5fcb34badc7f887525ba72805f2e480: Status 404 returned error can't find the container with id ad5aa22075be573c86f4011338a58ed5f5fcb34badc7f887525ba72805f2e480
Sep 15 07:54:09.827510 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.827469    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9" event={"ID":"d8000dd9-435e-4444-9a3c-5cfd153b74c6","Type":"ContainerStarted","Data":"4d4812ed58e13bc94b65b81f7c831b484f630d364ada7f608bb1f94a40fcee31"}
Sep 15 07:54:09.827681 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.827535    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9" event={"ID":"d8000dd9-435e-4444-9a3c-5cfd153b74c6","Type":"ContainerStarted","Data":"10a306f4d95a4a671131d46083a4a752c77aa5a7c4340827b07e473a81927dfc"}
Sep 15 07:54:09.827681 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.827653    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9"
Sep 15 07:54:09.829181 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.829158    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb" event={"ID":"ab4409e7-24c0-465b-b4f0-97481a9d5391","Type":"ContainerStarted","Data":"56658ee3e64e0b96176ca9b7f6e375c5979678975c38a0cf2dc9b88bedc7130c"}
Sep 15 07:54:09.829313 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.829193    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb" event={"ID":"ab4409e7-24c0-465b-b4f0-97481a9d5391","Type":"ContainerStarted","Data":"ad5aa22075be573c86f4011338a58ed5f5fcb34badc7f887525ba72805f2e480"}
Sep 15 07:54:09.829313 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.829297    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb"
Sep 15 07:54:09.844975 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.844931    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9" podStartSLOduration=0.844916986 podStartE2EDuration="844.916986ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:54:09.841934 +0000 UTC m=+2154.388504135" watchObservedRunningTime="2026-09-15 07:54:09.844916986 +0000 UTC m=+2154.391487084"
Sep 15 07:54:09.858733 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:09.858672    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb" podStartSLOduration=0.858657806 podStartE2EDuration="858.657806ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:54:09.856078336 +0000 UTC m=+2154.402648460" watchObservedRunningTime="2026-09-15 07:54:09.858657806 +0000 UTC m=+2154.405227910"
Sep 15 07:54:16.934239 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:16.934192    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp" podUID="ebf281d2-ecce-4ad9-95b5-383684a662f6" containerName="main" containerID="cri-o://c923af19ec8f6c8e0f100b61d1db8c53d5231cbbdf511ebbb3a82cf3c91fa5ab" gracePeriod=45
Sep 15 07:54:17.190588 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.190529    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:54:17.294678 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.294641    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-tmp-dir\") pod \"ebf281d2-ecce-4ad9-95b5-383684a662f6\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") "
Sep 15 07:54:17.294678 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.294679    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-model-cache\") pod \"ebf281d2-ecce-4ad9-95b5-383684a662f6\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") "
Sep 15 07:54:17.294941 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.294697    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-home\") pod \"ebf281d2-ecce-4ad9-95b5-383684a662f6\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") "
Sep 15 07:54:17.294941 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.294715    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-dshm\") pod \"ebf281d2-ecce-4ad9-95b5-383684a662f6\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") "
Sep 15 07:54:17.294941 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.294747    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf281d2-ecce-4ad9-95b5-383684a662f6-tls-certs\") pod \"ebf281d2-ecce-4ad9-95b5-383684a662f6\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") "
Sep 15 07:54:17.294941 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.294777    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4qz6\" (UniqueName: \"kubernetes.io/projected/ebf281d2-ecce-4ad9-95b5-383684a662f6-kube-api-access-k4qz6\") pod \"ebf281d2-ecce-4ad9-95b5-383684a662f6\" (UID: \"ebf281d2-ecce-4ad9-95b5-383684a662f6\") "
Sep 15 07:54:17.295167 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.294959    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ebf281d2-ecce-4ad9-95b5-383684a662f6" (UID: "ebf281d2-ecce-4ad9-95b5-383684a662f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:54:17.295167 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.294976    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "ebf281d2-ecce-4ad9-95b5-383684a662f6" (UID: "ebf281d2-ecce-4ad9-95b5-383684a662f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:54:17.295167 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.294952    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-home" (OuterVolumeSpecName: "home") pod "ebf281d2-ecce-4ad9-95b5-383684a662f6" (UID: "ebf281d2-ecce-4ad9-95b5-383684a662f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:54:17.295167 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.295133    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-tmp-dir\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:54:17.295167 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.295153    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-model-cache\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:54:17.295167 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.295166    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-home\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:54:17.296914 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.296890    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-dshm" (OuterVolumeSpecName: "dshm") pod "ebf281d2-ecce-4ad9-95b5-383684a662f6" (UID: "ebf281d2-ecce-4ad9-95b5-383684a662f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:54:17.296914 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.296898    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf281d2-ecce-4ad9-95b5-383684a662f6-kube-api-access-k4qz6" (OuterVolumeSpecName: "kube-api-access-k4qz6") pod "ebf281d2-ecce-4ad9-95b5-383684a662f6" (UID: "ebf281d2-ecce-4ad9-95b5-383684a662f6"). InnerVolumeSpecName "kube-api-access-k4qz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:54:17.297054 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.296899    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebf281d2-ecce-4ad9-95b5-383684a662f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ebf281d2-ecce-4ad9-95b5-383684a662f6" (UID: "ebf281d2-ecce-4ad9-95b5-383684a662f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:54:17.396631 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.396588    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf281d2-ecce-4ad9-95b5-383684a662f6-dshm\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:54:17.396631 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.396622    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf281d2-ecce-4ad9-95b5-383684a662f6-tls-certs\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:54:17.396631 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.396633    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4qz6\" (UniqueName: \"kubernetes.io/projected/ebf281d2-ecce-4ad9-95b5-383684a662f6-kube-api-access-k4qz6\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:54:17.872941 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.872897    2571 generic.go:356] "Generic (PLEG): container finished" podID="ebf281d2-ecce-4ad9-95b5-383684a662f6" containerID="c923af19ec8f6c8e0f100b61d1db8c53d5231cbbdf511ebbb3a82cf3c91fa5ab" exitCode=0
Sep 15 07:54:17.873152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.872946    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"
Sep 15 07:54:17.873152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.872978    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp" event={"ID":"ebf281d2-ecce-4ad9-95b5-383684a662f6","Type":"ContainerDied","Data":"c923af19ec8f6c8e0f100b61d1db8c53d5231cbbdf511ebbb3a82cf3c91fa5ab"}
Sep 15 07:54:17.873152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.873014    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp" event={"ID":"ebf281d2-ecce-4ad9-95b5-383684a662f6","Type":"ContainerDied","Data":"20899c995af88f0680531fddde3c79dafe80cc71d18cb5e08a81d763ea11d848"}
Sep 15 07:54:17.873152 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.873031    2571 scope.go:117] "RemoveContainer" containerID="c923af19ec8f6c8e0f100b61d1db8c53d5231cbbdf511ebbb3a82cf3c91fa5ab"
Sep 15 07:54:17.882739 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.882720    2571 scope.go:117] "RemoveContainer" containerID="c923af19ec8f6c8e0f100b61d1db8c53d5231cbbdf511ebbb3a82cf3c91fa5ab"
Sep 15 07:54:17.883000 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:54:17.882980    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c923af19ec8f6c8e0f100b61d1db8c53d5231cbbdf511ebbb3a82cf3c91fa5ab\": container with ID starting with c923af19ec8f6c8e0f100b61d1db8c53d5231cbbdf511ebbb3a82cf3c91fa5ab not found: ID does not exist" containerID="c923af19ec8f6c8e0f100b61d1db8c53d5231cbbdf511ebbb3a82cf3c91fa5ab"
Sep 15 07:54:17.883052 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.883007    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c923af19ec8f6c8e0f100b61d1db8c53d5231cbbdf511ebbb3a82cf3c91fa5ab"} err="failed to get container status \"c923af19ec8f6c8e0f100b61d1db8c53d5231cbbdf511ebbb3a82cf3c91fa5ab\": rpc error: code = NotFound desc = could not find container \"c923af19ec8f6c8e0f100b61d1db8c53d5231cbbdf511ebbb3a82cf3c91fa5ab\": container with ID starting with c923af19ec8f6c8e0f100b61d1db8c53d5231cbbdf511ebbb3a82cf3c91fa5ab not found: ID does not exist"
Sep 15 07:54:17.894695 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.894669    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"]
Sep 15 07:54:17.911211 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:17.911186    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-04464b/rate-test-model-04464b-kserve-9bb57c999-29xsp"]
Sep 15 07:54:19.910563 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:19.910532    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebf281d2-ecce-4ad9-95b5-383684a662f6" path="/var/lib/kubelet/pods/ebf281d2-ecce-4ad9-95b5-383684a662f6/volumes"
Sep 15 07:54:20.836652 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:20.836620    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb"
Sep 15 07:54:20.836855 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:20.836691    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9"
Sep 15 07:54:39.571781 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.571743    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q"]
Sep 15 07:54:39.574174 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.572196    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebf281d2-ecce-4ad9-95b5-383684a662f6" containerName="main"
Sep 15 07:54:39.574174 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.572207    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebf281d2-ecce-4ad9-95b5-383684a662f6" containerName="main"
Sep 15 07:54:39.574174 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.572302    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebf281d2-ecce-4ad9-95b5-383684a662f6" containerName="main"
Sep 15 07:54:39.575172 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.575156    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q"
Sep 15 07:54:39.577780 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.577758    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5d58bd-dockercfg-xvnwn\""
Sep 15 07:54:39.578024 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.578005    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5d58bd\""
Sep 15 07:54:39.584890 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.584869    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q"]
Sep 15 07:54:39.631677 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.631640    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58"]
Sep 15 07:54:39.635377 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.635360    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58"
Sep 15 07:54:39.642651 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.642618    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58"]
Sep 15 07:54:39.709959 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.709917    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qlvw\" (UniqueName: \"kubernetes.io/projected/c2080dfb-4d30-41b7-8fd8-2f2af167bc88-kube-api-access-9qlvw\") pod \"payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q\" (UID: \"c2080dfb-4d30-41b7-8fd8-2f2af167bc88\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q"
Sep 15 07:54:39.710134 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.709971    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2080dfb-4d30-41b7-8fd8-2f2af167bc88-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q\" (UID: \"c2080dfb-4d30-41b7-8fd8-2f2af167bc88\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q"
Sep 15 07:54:39.811159 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.811122    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34047e84-9f7a-40e3-bc86-a70880df93b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58\" (UID: \"34047e84-9f7a-40e3-bc86-a70880df93b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58"
Sep 15 07:54:39.811366 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.811197    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qlvw\" (UniqueName: \"kubernetes.io/projected/c2080dfb-4d30-41b7-8fd8-2f2af167bc88-kube-api-access-9qlvw\") pod \"payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q\" (UID: \"c2080dfb-4d30-41b7-8fd8-2f2af167bc88\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q"
Sep 15 07:54:39.811366 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.811230    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2080dfb-4d30-41b7-8fd8-2f2af167bc88-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q\" (UID: \"c2080dfb-4d30-41b7-8fd8-2f2af167bc88\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q"
Sep 15 07:54:39.811366 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.811308    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8skl\" (UniqueName: \"kubernetes.io/projected/34047e84-9f7a-40e3-bc86-a70880df93b9-kube-api-access-f8skl\") pod \"payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58\" (UID: \"34047e84-9f7a-40e3-bc86-a70880df93b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58"
Sep 15 07:54:39.812031 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.812004    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2080dfb-4d30-41b7-8fd8-2f2af167bc88-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q\" (UID: \"c2080dfb-4d30-41b7-8fd8-2f2af167bc88\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q"
Sep 15 07:54:39.818696 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.818674    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qlvw\" (UniqueName: \"kubernetes.io/projected/c2080dfb-4d30-41b7-8fd8-2f2af167bc88-kube-api-access-9qlvw\") pod \"payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q\" (UID: \"c2080dfb-4d30-41b7-8fd8-2f2af167bc88\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q"
Sep 15 07:54:39.887196 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.887117    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q"
Sep 15 07:54:39.912110 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.912077    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34047e84-9f7a-40e3-bc86-a70880df93b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58\" (UID: \"34047e84-9f7a-40e3-bc86-a70880df93b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58"
Sep 15 07:54:39.912290 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.912188    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8skl\" (UniqueName: \"kubernetes.io/projected/34047e84-9f7a-40e3-bc86-a70880df93b9-kube-api-access-f8skl\") pod \"payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58\" (UID: \"34047e84-9f7a-40e3-bc86-a70880df93b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58"
Sep 15 07:54:39.912963 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.912942    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34047e84-9f7a-40e3-bc86-a70880df93b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58\" (UID: \"34047e84-9f7a-40e3-bc86-a70880df93b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58"
Sep 15 07:54:39.919720 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.919697    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8skl\" (UniqueName: \"kubernetes.io/projected/34047e84-9f7a-40e3-bc86-a70880df93b9-kube-api-access-f8skl\") pod \"payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58\" (UID: \"34047e84-9f7a-40e3-bc86-a70880df93b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58"
Sep 15 07:54:39.947686 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:39.947644    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58"
Sep 15 07:54:40.162498 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:40.162463    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q"]
Sep 15 07:54:40.164188 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:54:40.164159    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2080dfb_4d30_41b7_8fd8_2f2af167bc88.slice/crio-3cb40e0534b19e38e6ed573ba3ece26f36b5ad0580a5241bc95f2478d29bb536 WatchSource:0}: Error finding container 3cb40e0534b19e38e6ed573ba3ece26f36b5ad0580a5241bc95f2478d29bb536: Status 404 returned error can't find the container with id 3cb40e0534b19e38e6ed573ba3ece26f36b5ad0580a5241bc95f2478d29bb536
Sep 15 07:54:40.225379 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:40.225353    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58"]
Sep 15 07:54:40.226708 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:54:40.226684    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34047e84_9f7a_40e3_bc86_a70880df93b9.slice/crio-84923298798de1a3b0abfd9992090d74f1f28d70700896c0663081a3a115eb94 WatchSource:0}: Error finding container 84923298798de1a3b0abfd9992090d74f1f28d70700896c0663081a3a115eb94: Status 404 returned error can't find the container with id 84923298798de1a3b0abfd9992090d74f1f28d70700896c0663081a3a115eb94
Sep 15 07:54:41.000953 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:41.000906    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58" event={"ID":"34047e84-9f7a-40e3-bc86-a70880df93b9","Type":"ContainerStarted","Data":"d13c7b7d76577741b501d647345525dad2196835a470eb85be8de54468002d70"}
Sep 15 07:54:41.001430 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:41.000970    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58" event={"ID":"34047e84-9f7a-40e3-bc86-a70880df93b9","Type":"ContainerStarted","Data":"84923298798de1a3b0abfd9992090d74f1f28d70700896c0663081a3a115eb94"}
Sep 15 07:54:41.001430 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:41.001004    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58"
Sep 15 07:54:41.002591 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:41.002562    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q" event={"ID":"c2080dfb-4d30-41b7-8fd8-2f2af167bc88","Type":"ContainerStarted","Data":"cc0f5db975474fa40f56775629756d287124c2786e55c77f78193f0a3b8928d4"}
Sep 15 07:54:41.002669 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:41.002615    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q" event={"ID":"c2080dfb-4d30-41b7-8fd8-2f2af167bc88","Type":"ContainerStarted","Data":"3cb40e0534b19e38e6ed573ba3ece26f36b5ad0580a5241bc95f2478d29bb536"}
Sep 15 07:54:41.002669 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:41.002644    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q"
Sep 15 07:54:41.030283 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:41.030213    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58" podStartSLOduration=2.030200542 podStartE2EDuration="2.030200542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:54:41.029297554 +0000 UTC m=+2185.575867655" watchObservedRunningTime="2026-09-15 07:54:41.030200542 +0000 UTC m=+2185.576770640"
Sep 15 07:54:41.056792 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:41.056743    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q" podStartSLOduration=2.056727336 podStartE2EDuration="2.056727336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:54:41.055070761 +0000 UTC m=+2185.601640873" watchObservedRunningTime="2026-09-15 07:54:41.056727336 +0000 UTC m=+2185.603297436"
Sep 15 07:54:52.011073 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:52.011039    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58"
Sep 15 07:54:52.011564 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:54:52.011108    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q"
Sep 15 07:55:11.149315 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:55:11.149278    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q_c2080dfb-4d30-41b7-8fd8-2f2af167bc88/payload-processing/0.log"
Sep 15 07:55:16.291045 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:55:16.291007    2571 scope.go:117] "RemoveContainer" containerID="3380264c958dc6bdb9d1948ba0ac18cdebdec15e464886ec70aa8b1687e02dc2"
Sep 15 07:55:44.118418 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:55:44.118386    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb_ab4409e7-24c0-465b-b4f0-97481a9d5391/payload-processing/0.log"
Sep 15 07:55:44.353296 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:55:44.353250    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q_c2080dfb-4d30-41b7-8fd8-2f2af167bc88/payload-processing/0.log"
Sep 15 07:55:52.632000 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:55:52.631962    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm"]
Sep 15 07:55:52.635734 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:55:52.635715    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm"
Sep 15 07:55:52.638181 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:55:52.638153    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-0fd0de-dockercfg-7plw7\""
Sep 15 07:55:52.638307 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:55:52.638162    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-0fd0de\""
Sep 15 07:55:52.646525 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:55:52.646498    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm"]
Sep 15 07:55:52.713349 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:55:52.713314    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h4hf\" (UniqueName: \"kubernetes.io/projected/b74ef3ed-e919-46b0-83a5-6823ef86735b-kube-api-access-2h4hf\") pod \"payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm\" (UID: \"b74ef3ed-e919-46b0-83a5-6823ef86735b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm"
Sep 15 07:55:52.713523 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:55:52.713369    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b74ef3ed-e919-46b0-83a5-6823ef86735b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm\" (UID: \"b74ef3ed-e919-46b0-83a5-6823ef86735b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm"
Sep 15 07:55:52.813841 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:55:52.813803    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b74ef3ed-e919-46b0-83a5-6823ef86735b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm\" (UID: \"b74ef3ed-e919-46b0-83a5-6823ef86735b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm"
Sep 15 07:55:52.814020 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:55:52.813905    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h4hf\" (UniqueName: \"kubernetes.io/projected/b74ef3ed-e919-46b0-83a5-6823ef86735b-kube-api-access-2h4hf\") pod \"payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm\" (UID: \"b74ef3ed-e919-46b0-83a5-6823ef86735b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm"
Sep 15 07:55:52.814683 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:55:52.814663    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b74ef3ed-e919-46b0-83a5-6823ef86735b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm\" (UID: \"b74ef3ed-e919-46b0-83a5-6823ef86735b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm"
Sep 15 07:55:52.822366 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:55:52.822335    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h4hf\" (UniqueName: \"kubernetes.io/projected/b74ef3ed-e919-46b0-83a5-6823ef86735b-kube-api-access-2h4hf\") pod \"payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm\" (UID: \"b74ef3ed-e919-46b0-83a5-6823ef86735b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm"
Sep 15 07:55:52.947176 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:55:52.947088    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm"
Sep 15 07:55:53.220422 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:55:53.220388    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm"]
Sep 15 07:55:53.223969 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:55:53.223940    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74ef3ed_e919_46b0_83a5_6823ef86735b.slice/crio-9c28907a72241196f76eb18f42209b2b65a5cf64848225f0059125554357e040 WatchSource:0}: Error finding container 9c28907a72241196f76eb18f42209b2b65a5cf64848225f0059125554357e040: Status 404 returned error can't find the container with id 9c28907a72241196f76eb18f42209b2b65a5cf64848225f0059125554357e040
Sep 15 07:55:53.388403 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:55:53.388354    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm" event={"ID":"b74ef3ed-e919-46b0-83a5-6823ef86735b","Type":"ContainerStarted","Data":"05e96214216515672be7e4dcf43684445f785ddba8d56bd5dbba4b2f819eedb4"}
Sep 15 07:55:53.388575 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:55:53.388427    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm" event={"ID":"b74ef3ed-e919-46b0-83a5-6823ef86735b","Type":"ContainerStarted","Data":"9c28907a72241196f76eb18f42209b2b65a5cf64848225f0059125554357e040"}
Sep 15 07:55:53.388629 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:55:53.388580    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm"
Sep 15 07:55:53.403947 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:55:53.403890    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm" podStartSLOduration=1.403875142 podStartE2EDuration="1.403875142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:55:53.40300696 +0000 UTC m=+2257.949577079" watchObservedRunningTime="2026-09-15 07:55:53.403875142 +0000 UTC m=+2257.950445238"
Sep 15 07:56:04.396980 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:04.396941    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm"
Sep 15 07:56:19.140683 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:19.140645    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0fd0de-d2xwt"]
Sep 15 07:56:19.144579 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:19.144560    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0fd0de-d2xwt"
Sep 15 07:56:19.146836 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:19.146812    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 07:56:19.146836 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:19.146827    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8jpwk\""
Sep 15 07:56:19.153107 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:19.153071    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0fd0de-d2xwt"]
Sep 15 07:56:19.270059 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:19.270022    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24534567-6ef4-4984-8aa8-436f29eb917f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-0fd0de-d2xwt\" (UID: \"24534567-6ef4-4984-8aa8-436f29eb917f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0fd0de-d2xwt"
Sep 15 07:56:19.371148 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:19.371109    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24534567-6ef4-4984-8aa8-436f29eb917f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-0fd0de-d2xwt\" (UID: \"24534567-6ef4-4984-8aa8-436f29eb917f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0fd0de-d2xwt"
Sep 15 07:56:19.371775 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:19.371753    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24534567-6ef4-4984-8aa8-436f29eb917f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-0fd0de-d2xwt\" (UID: \"24534567-6ef4-4984-8aa8-436f29eb917f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0fd0de-d2xwt"
Sep 15 07:56:19.456769 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:19.456733    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0fd0de-d2xwt"
Sep 15 07:56:19.726614 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:19.726578    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0fd0de-d2xwt"]
Sep 15 07:56:19.728013 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:56:19.727982    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24534567_6ef4_4984_8aa8_436f29eb917f.slice/crio-72aa75031f0ec37d31dba6b08fcc0e8e50c05e6d79ce54d7940b6931749a05c9 WatchSource:0}: Error finding container 72aa75031f0ec37d31dba6b08fcc0e8e50c05e6d79ce54d7940b6931749a05c9: Status 404 returned error can't find the container with id 72aa75031f0ec37d31dba6b08fcc0e8e50c05e6d79ce54d7940b6931749a05c9
Sep 15 07:56:20.542083 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:20.541983    2571 generic.go:356] "Generic (PLEG): container finished" podID="24534567-6ef4-4984-8aa8-436f29eb917f" containerID="76b190fbefb1cd121d39910b70ab43396c0a55bda9a8591409d9f57ef5e85e10" exitCode=0
Sep 15 07:56:20.542083 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:20.542047    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0fd0de-d2xwt" event={"ID":"24534567-6ef4-4984-8aa8-436f29eb917f","Type":"ContainerDied","Data":"76b190fbefb1cd121d39910b70ab43396c0a55bda9a8591409d9f57ef5e85e10"}
Sep 15 07:56:20.542673 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:20.542102    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0fd0de-d2xwt" event={"ID":"24534567-6ef4-4984-8aa8-436f29eb917f","Type":"ContainerStarted","Data":"72aa75031f0ec37d31dba6b08fcc0e8e50c05e6d79ce54d7940b6931749a05c9"}
Sep 15 07:56:21.683919 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:21.683894    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0fd0de-d2xwt"
Sep 15 07:56:21.796878 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:21.796847    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24534567-6ef4-4984-8aa8-436f29eb917f-maas-api-service-ca\") pod \"24534567-6ef4-4984-8aa8-436f29eb917f\" (UID: \"24534567-6ef4-4984-8aa8-436f29eb917f\") "
Sep 15 07:56:21.797186 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:21.797165    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24534567-6ef4-4984-8aa8-436f29eb917f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "24534567-6ef4-4984-8aa8-436f29eb917f" (UID: "24534567-6ef4-4984-8aa8-436f29eb917f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:56:21.898613 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:21.898537    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24534567-6ef4-4984-8aa8-436f29eb917f-maas-api-service-ca\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:56:22.553996 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:22.553963    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0fd0de-d2xwt" event={"ID":"24534567-6ef4-4984-8aa8-436f29eb917f","Type":"ContainerDied","Data":"72aa75031f0ec37d31dba6b08fcc0e8e50c05e6d79ce54d7940b6931749a05c9"}
Sep 15 07:56:22.554168 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:22.554010    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72aa75031f0ec37d31dba6b08fcc0e8e50c05e6d79ce54d7940b6931749a05c9"
Sep 15 07:56:22.554168 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:22.553977    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0fd0de-d2xwt"
Sep 15 07:56:30.139714 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:30.139679    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm"]
Sep 15 07:56:30.140140 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:30.139899    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm" podUID="b74ef3ed-e919-46b0-83a5-6823ef86735b" containerName="payload-pre-processing" containerID="cri-o://05e96214216515672be7e4dcf43684445f785ddba8d56bd5dbba4b2f819eedb4" gracePeriod=30
Sep 15 07:56:30.319567 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:30.319496    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0fd0de-d2xwt"]
Sep 15 07:56:30.322596 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:30.322548    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0fd0de-d2xwt"]
Sep 15 07:56:30.399200 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:30.399144    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm"
Sep 15 07:56:30.486133 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:30.486094    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h4hf\" (UniqueName: \"kubernetes.io/projected/b74ef3ed-e919-46b0-83a5-6823ef86735b-kube-api-access-2h4hf\") pod \"b74ef3ed-e919-46b0-83a5-6823ef86735b\" (UID: \"b74ef3ed-e919-46b0-83a5-6823ef86735b\") "
Sep 15 07:56:30.486360 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:30.486155    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b74ef3ed-e919-46b0-83a5-6823ef86735b-plugins-config-volume\") pod \"b74ef3ed-e919-46b0-83a5-6823ef86735b\" (UID: \"b74ef3ed-e919-46b0-83a5-6823ef86735b\") "
Sep 15 07:56:30.486604 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:30.486579    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b74ef3ed-e919-46b0-83a5-6823ef86735b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b74ef3ed-e919-46b0-83a5-6823ef86735b" (UID: "b74ef3ed-e919-46b0-83a5-6823ef86735b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:56:30.488230 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:30.488206    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74ef3ed-e919-46b0-83a5-6823ef86735b-kube-api-access-2h4hf" (OuterVolumeSpecName: "kube-api-access-2h4hf") pod "b74ef3ed-e919-46b0-83a5-6823ef86735b" (UID: "b74ef3ed-e919-46b0-83a5-6823ef86735b"). InnerVolumeSpecName "kube-api-access-2h4hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:56:30.587354 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:30.587319    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h4hf\" (UniqueName: \"kubernetes.io/projected/b74ef3ed-e919-46b0-83a5-6823ef86735b-kube-api-access-2h4hf\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:56:30.587354 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:30.587348    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b74ef3ed-e919-46b0-83a5-6823ef86735b-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:56:30.596871 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:30.596835    2571 generic.go:356] "Generic (PLEG): container finished" podID="b74ef3ed-e919-46b0-83a5-6823ef86735b" containerID="05e96214216515672be7e4dcf43684445f785ddba8d56bd5dbba4b2f819eedb4" exitCode=0
Sep 15 07:56:30.597029 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:30.596887    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm"
Sep 15 07:56:30.597029 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:30.596925    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm" event={"ID":"b74ef3ed-e919-46b0-83a5-6823ef86735b","Type":"ContainerDied","Data":"05e96214216515672be7e4dcf43684445f785ddba8d56bd5dbba4b2f819eedb4"}
Sep 15 07:56:30.597029 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:30.596982    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm" event={"ID":"b74ef3ed-e919-46b0-83a5-6823ef86735b","Type":"ContainerDied","Data":"9c28907a72241196f76eb18f42209b2b65a5cf64848225f0059125554357e040"}
Sep 15 07:56:30.597029 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:30.597012    2571 scope.go:117] "RemoveContainer" containerID="05e96214216515672be7e4dcf43684445f785ddba8d56bd5dbba4b2f819eedb4"
Sep 15 07:56:30.606754 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:30.606734    2571 scope.go:117] "RemoveContainer" containerID="05e96214216515672be7e4dcf43684445f785ddba8d56bd5dbba4b2f819eedb4"
Sep 15 07:56:30.607012 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:56:30.606995    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05e96214216515672be7e4dcf43684445f785ddba8d56bd5dbba4b2f819eedb4\": container with ID starting with 05e96214216515672be7e4dcf43684445f785ddba8d56bd5dbba4b2f819eedb4 not found: ID does not exist" containerID="05e96214216515672be7e4dcf43684445f785ddba8d56bd5dbba4b2f819eedb4"
Sep 15 07:56:30.607065 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:30.607019    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05e96214216515672be7e4dcf43684445f785ddba8d56bd5dbba4b2f819eedb4"} err="failed to get container status \"05e96214216515672be7e4dcf43684445f785ddba8d56bd5dbba4b2f819eedb4\": rpc error: code = NotFound desc = could not find container \"05e96214216515672be7e4dcf43684445f785ddba8d56bd5dbba4b2f819eedb4\": container with ID starting with 05e96214216515672be7e4dcf43684445f785ddba8d56bd5dbba4b2f819eedb4 not found: ID does not exist"
Sep 15 07:56:30.618146 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:30.618120    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm"]
Sep 15 07:56:30.624530 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:30.624506    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0fd0de-68c5d96874-wzzfm"]
Sep 15 07:56:31.909769 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:31.909734    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24534567-6ef4-4984-8aa8-436f29eb917f" path="/var/lib/kubelet/pods/24534567-6ef4-4984-8aa8-436f29eb917f/volumes"
Sep 15 07:56:31.910175 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:31.910159    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b74ef3ed-e919-46b0-83a5-6823ef86735b" path="/var/lib/kubelet/pods/b74ef3ed-e919-46b0-83a5-6823ef86735b/volumes"
Sep 15 07:56:40.409834 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:40.409803    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b179ed-ffg7d"]
Sep 15 07:56:40.410272 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:40.410243    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b74ef3ed-e919-46b0-83a5-6823ef86735b" containerName="payload-pre-processing"
Sep 15 07:56:40.410272 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:40.410270    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b74ef3ed-e919-46b0-83a5-6823ef86735b" containerName="payload-pre-processing"
Sep 15 07:56:40.410356 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:40.410283    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24534567-6ef4-4984-8aa8-436f29eb917f" containerName="revoke-keys"
Sep 15 07:56:40.410356 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:40.410289    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="24534567-6ef4-4984-8aa8-436f29eb917f" containerName="revoke-keys"
Sep 15 07:56:40.410426 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:40.410383    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="24534567-6ef4-4984-8aa8-436f29eb917f" containerName="revoke-keys"
Sep 15 07:56:40.410426 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:40.410397    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b74ef3ed-e919-46b0-83a5-6823ef86735b" containerName="payload-pre-processing"
Sep 15 07:56:40.415381 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:40.415363    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b179ed-ffg7d"
Sep 15 07:56:40.417925 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:40.417899    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 07:56:40.418068 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:40.417932    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8jpwk\""
Sep 15 07:56:40.421860 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:40.421828    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b179ed-ffg7d"]
Sep 15 07:56:40.482050 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:40.482010    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/395d9fa3-23fa-4940-80ea-7c5d77dd73ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b179ed-ffg7d\" (UID: \"395d9fa3-23fa-4940-80ea-7c5d77dd73ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b179ed-ffg7d"
Sep 15 07:56:40.583175 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:40.583132    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/395d9fa3-23fa-4940-80ea-7c5d77dd73ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b179ed-ffg7d\" (UID: \"395d9fa3-23fa-4940-80ea-7c5d77dd73ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b179ed-ffg7d"
Sep 15 07:56:40.583768 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:40.583747    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/395d9fa3-23fa-4940-80ea-7c5d77dd73ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b179ed-ffg7d\" (UID: \"395d9fa3-23fa-4940-80ea-7c5d77dd73ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b179ed-ffg7d"
Sep 15 07:56:40.728014 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:40.727979    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b179ed-ffg7d"
Sep 15 07:56:41.000915 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:41.000890    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b179ed-ffg7d"]
Sep 15 07:56:41.002987 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:56:41.002961    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395d9fa3_23fa_4940_80ea_7c5d77dd73ac.slice/crio-20a316ad16770078f4158a912d5d0afa0b14d54a3241419e09206078f4b15b3d WatchSource:0}: Error finding container 20a316ad16770078f4158a912d5d0afa0b14d54a3241419e09206078f4b15b3d: Status 404 returned error can't find the container with id 20a316ad16770078f4158a912d5d0afa0b14d54a3241419e09206078f4b15b3d
Sep 15 07:56:41.659469 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:41.659377    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b179ed-ffg7d" event={"ID":"395d9fa3-23fa-4940-80ea-7c5d77dd73ac","Type":"ContainerStarted","Data":"2274ce7a25bb299fe196261ed50b09dfedde596ebc678b65850b892495b15190"}
Sep 15 07:56:41.659469 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:41.659441    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b179ed-ffg7d" event={"ID":"395d9fa3-23fa-4940-80ea-7c5d77dd73ac","Type":"ContainerStarted","Data":"20a316ad16770078f4158a912d5d0afa0b14d54a3241419e09206078f4b15b3d"}
Sep 15 07:56:41.678814 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:41.678758    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b179ed-ffg7d" podStartSLOduration=1.6787424039999999 podStartE2EDuration="1.678742404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:56:41.677039999 +0000 UTC m=+2306.223610097" watchObservedRunningTime="2026-09-15 07:56:41.678742404 +0000 UTC m=+2306.225312546"
Sep 15 07:56:42.666301 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:42.666245    2571 generic.go:356] "Generic (PLEG): container finished" podID="395d9fa3-23fa-4940-80ea-7c5d77dd73ac" containerID="2274ce7a25bb299fe196261ed50b09dfedde596ebc678b65850b892495b15190" exitCode=0
Sep 15 07:56:42.666708 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:42.666345    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b179ed-ffg7d" event={"ID":"395d9fa3-23fa-4940-80ea-7c5d77dd73ac","Type":"ContainerDied","Data":"2274ce7a25bb299fe196261ed50b09dfedde596ebc678b65850b892495b15190"}
Sep 15 07:56:43.807871 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:43.807842    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b179ed-ffg7d"
Sep 15 07:56:43.908852 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:43.908824    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/395d9fa3-23fa-4940-80ea-7c5d77dd73ac-maas-api-service-ca\") pod \"395d9fa3-23fa-4940-80ea-7c5d77dd73ac\" (UID: \"395d9fa3-23fa-4940-80ea-7c5d77dd73ac\") "
Sep 15 07:56:43.909193 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:43.909169    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/395d9fa3-23fa-4940-80ea-7c5d77dd73ac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "395d9fa3-23fa-4940-80ea-7c5d77dd73ac" (UID: "395d9fa3-23fa-4940-80ea-7c5d77dd73ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:56:44.009777 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:44.009739    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/395d9fa3-23fa-4940-80ea-7c5d77dd73ac-maas-api-service-ca\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:56:44.681688 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:44.681645    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b179ed-ffg7d"
Sep 15 07:56:44.681688 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:44.681671    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b179ed-ffg7d" event={"ID":"395d9fa3-23fa-4940-80ea-7c5d77dd73ac","Type":"ContainerDied","Data":"20a316ad16770078f4158a912d5d0afa0b14d54a3241419e09206078f4b15b3d"}
Sep 15 07:56:44.681925 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:44.681723    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20a316ad16770078f4158a912d5d0afa0b14d54a3241419e09206078f4b15b3d"
Sep 15 07:56:56.043453 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.043415    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb"]
Sep 15 07:56:56.044046 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.043626    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb" podUID="ab4409e7-24c0-465b-b4f0-97481a9d5391" containerName="payload-processing" containerID="cri-o://56658ee3e64e0b96176ca9b7f6e375c5979678975c38a0cf2dc9b88bedc7130c" gracePeriod=30
Sep 15 07:56:56.052647 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.052612    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9"]
Sep 15 07:56:56.052905 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.052858    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9" podUID="d8000dd9-435e-4444-9a3c-5cfd153b74c6" containerName="payload-pre-processing" containerID="cri-o://4d4812ed58e13bc94b65b81f7c831b484f630d364ada7f608bb1f94a40fcee31" gracePeriod=30
Sep 15 07:56:56.230999 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.230970    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b179ed-ffg7d"]
Sep 15 07:56:56.232895 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.232870    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b179ed-ffg7d"]
Sep 15 07:56:56.321479 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.321452    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb"
Sep 15 07:56:56.337621 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.337594    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9"
Sep 15 07:56:56.431645 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.431607    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vvkx\" (UniqueName: \"kubernetes.io/projected/d8000dd9-435e-4444-9a3c-5cfd153b74c6-kube-api-access-2vvkx\") pod \"d8000dd9-435e-4444-9a3c-5cfd153b74c6\" (UID: \"d8000dd9-435e-4444-9a3c-5cfd153b74c6\") "
Sep 15 07:56:56.431849 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.431653    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8000dd9-435e-4444-9a3c-5cfd153b74c6-plugins-config-volume\") pod \"d8000dd9-435e-4444-9a3c-5cfd153b74c6\" (UID: \"d8000dd9-435e-4444-9a3c-5cfd153b74c6\") "
Sep 15 07:56:56.431849 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.431687    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dqtq\" (UniqueName: \"kubernetes.io/projected/ab4409e7-24c0-465b-b4f0-97481a9d5391-kube-api-access-5dqtq\") pod \"ab4409e7-24c0-465b-b4f0-97481a9d5391\" (UID: \"ab4409e7-24c0-465b-b4f0-97481a9d5391\") "
Sep 15 07:56:56.431849 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.431815    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4409e7-24c0-465b-b4f0-97481a9d5391-plugins-config-volume\") pod \"ab4409e7-24c0-465b-b4f0-97481a9d5391\" (UID: \"ab4409e7-24c0-465b-b4f0-97481a9d5391\") "
Sep 15 07:56:56.432095 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.432070    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8000dd9-435e-4444-9a3c-5cfd153b74c6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d8000dd9-435e-4444-9a3c-5cfd153b74c6" (UID: "d8000dd9-435e-4444-9a3c-5cfd153b74c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:56:56.432191 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.432163    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab4409e7-24c0-465b-b4f0-97481a9d5391-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ab4409e7-24c0-465b-b4f0-97481a9d5391" (UID: "ab4409e7-24c0-465b-b4f0-97481a9d5391"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:56:56.432340 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.432326    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8000dd9-435e-4444-9a3c-5cfd153b74c6-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:56:56.432392 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.432343    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4409e7-24c0-465b-b4f0-97481a9d5391-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:56:56.433805 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.433785    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8000dd9-435e-4444-9a3c-5cfd153b74c6-kube-api-access-2vvkx" (OuterVolumeSpecName: "kube-api-access-2vvkx") pod "d8000dd9-435e-4444-9a3c-5cfd153b74c6" (UID: "d8000dd9-435e-4444-9a3c-5cfd153b74c6"). InnerVolumeSpecName "kube-api-access-2vvkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:56:56.434013 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.433992    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4409e7-24c0-465b-b4f0-97481a9d5391-kube-api-access-5dqtq" (OuterVolumeSpecName: "kube-api-access-5dqtq") pod "ab4409e7-24c0-465b-b4f0-97481a9d5391" (UID: "ab4409e7-24c0-465b-b4f0-97481a9d5391"). InnerVolumeSpecName "kube-api-access-5dqtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:56:56.532890 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.532858    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dqtq\" (UniqueName: \"kubernetes.io/projected/ab4409e7-24c0-465b-b4f0-97481a9d5391-kube-api-access-5dqtq\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:56:56.532890 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.532886    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vvkx\" (UniqueName: \"kubernetes.io/projected/d8000dd9-435e-4444-9a3c-5cfd153b74c6-kube-api-access-2vvkx\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:56:56.748077 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.748038    2571 generic.go:356] "Generic (PLEG): container finished" podID="d8000dd9-435e-4444-9a3c-5cfd153b74c6" containerID="4d4812ed58e13bc94b65b81f7c831b484f630d364ada7f608bb1f94a40fcee31" exitCode=0
Sep 15 07:56:56.748077 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.748084    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9"
Sep 15 07:56:56.748311 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.748128    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9" event={"ID":"d8000dd9-435e-4444-9a3c-5cfd153b74c6","Type":"ContainerDied","Data":"4d4812ed58e13bc94b65b81f7c831b484f630d364ada7f608bb1f94a40fcee31"}
Sep 15 07:56:56.748311 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.748190    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9" event={"ID":"d8000dd9-435e-4444-9a3c-5cfd153b74c6","Type":"ContainerDied","Data":"10a306f4d95a4a671131d46083a4a752c77aa5a7c4340827b07e473a81927dfc"}
Sep 15 07:56:56.748311 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.748215    2571 scope.go:117] "RemoveContainer" containerID="4d4812ed58e13bc94b65b81f7c831b484f630d364ada7f608bb1f94a40fcee31"
Sep 15 07:56:56.749759 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.749732    2571 generic.go:356] "Generic (PLEG): container finished" podID="ab4409e7-24c0-465b-b4f0-97481a9d5391" containerID="56658ee3e64e0b96176ca9b7f6e375c5979678975c38a0cf2dc9b88bedc7130c" exitCode=0
Sep 15 07:56:56.749875 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.749767    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb"
Sep 15 07:56:56.749875 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.749770    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb" event={"ID":"ab4409e7-24c0-465b-b4f0-97481a9d5391","Type":"ContainerDied","Data":"56658ee3e64e0b96176ca9b7f6e375c5979678975c38a0cf2dc9b88bedc7130c"}
Sep 15 07:56:56.749875 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.749812    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb" event={"ID":"ab4409e7-24c0-465b-b4f0-97481a9d5391","Type":"ContainerDied","Data":"ad5aa22075be573c86f4011338a58ed5f5fcb34badc7f887525ba72805f2e480"}
Sep 15 07:56:56.758481 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.758465    2571 scope.go:117] "RemoveContainer" containerID="4d4812ed58e13bc94b65b81f7c831b484f630d364ada7f608bb1f94a40fcee31"
Sep 15 07:56:56.758789 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:56:56.758770    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d4812ed58e13bc94b65b81f7c831b484f630d364ada7f608bb1f94a40fcee31\": container with ID starting with 4d4812ed58e13bc94b65b81f7c831b484f630d364ada7f608bb1f94a40fcee31 not found: ID does not exist" containerID="4d4812ed58e13bc94b65b81f7c831b484f630d364ada7f608bb1f94a40fcee31"
Sep 15 07:56:56.758868 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.758796    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d4812ed58e13bc94b65b81f7c831b484f630d364ada7f608bb1f94a40fcee31"} err="failed to get container status \"4d4812ed58e13bc94b65b81f7c831b484f630d364ada7f608bb1f94a40fcee31\": rpc error: code = NotFound desc = could not find container \"4d4812ed58e13bc94b65b81f7c831b484f630d364ada7f608bb1f94a40fcee31\": container with ID starting with 4d4812ed58e13bc94b65b81f7c831b484f630d364ada7f608bb1f94a40fcee31 not found: ID does not exist"
Sep 15 07:56:56.758868 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.758811    2571 scope.go:117] "RemoveContainer" containerID="56658ee3e64e0b96176ca9b7f6e375c5979678975c38a0cf2dc9b88bedc7130c"
Sep 15 07:56:56.766888 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.766868    2571 scope.go:117] "RemoveContainer" containerID="56658ee3e64e0b96176ca9b7f6e375c5979678975c38a0cf2dc9b88bedc7130c"
Sep 15 07:56:56.767141 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:56:56.767124    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56658ee3e64e0b96176ca9b7f6e375c5979678975c38a0cf2dc9b88bedc7130c\": container with ID starting with 56658ee3e64e0b96176ca9b7f6e375c5979678975c38a0cf2dc9b88bedc7130c not found: ID does not exist" containerID="56658ee3e64e0b96176ca9b7f6e375c5979678975c38a0cf2dc9b88bedc7130c"
Sep 15 07:56:56.767183 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.767147    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56658ee3e64e0b96176ca9b7f6e375c5979678975c38a0cf2dc9b88bedc7130c"} err="failed to get container status \"56658ee3e64e0b96176ca9b7f6e375c5979678975c38a0cf2dc9b88bedc7130c\": rpc error: code = NotFound desc = could not find container \"56658ee3e64e0b96176ca9b7f6e375c5979678975c38a0cf2dc9b88bedc7130c\": container with ID starting with 56658ee3e64e0b96176ca9b7f6e375c5979678975c38a0cf2dc9b88bedc7130c not found: ID does not exist"
Sep 15 07:56:56.772919 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.772898    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb"]
Sep 15 07:56:56.778521 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.778501    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b179ed-68c45db669-zm5wb"]
Sep 15 07:56:56.787966 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.787944    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9"]
Sep 15 07:56:56.792134 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:56.792114    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b179ed-7cf54b5ff4-7k9x9"]
Sep 15 07:56:57.909295 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:57.909235    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="395d9fa3-23fa-4940-80ea-7c5d77dd73ac" path="/var/lib/kubelet/pods/395d9fa3-23fa-4940-80ea-7c5d77dd73ac/volumes"
Sep 15 07:56:57.909668 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:57.909644    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab4409e7-24c0-465b-b4f0-97481a9d5391" path="/var/lib/kubelet/pods/ab4409e7-24c0-465b-b4f0-97481a9d5391/volumes"
Sep 15 07:56:57.909970 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:56:57.909957    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8000dd9-435e-4444-9a3c-5cfd153b74c6" path="/var/lib/kubelet/pods/d8000dd9-435e-4444-9a3c-5cfd153b74c6/volumes"
Sep 15 07:57:44.897351 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:44.897311    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d58bd-qwvlr"]
Sep 15 07:57:44.897779 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:44.897719    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8000dd9-435e-4444-9a3c-5cfd153b74c6" containerName="payload-pre-processing"
Sep 15 07:57:44.897779 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:44.897730    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8000dd9-435e-4444-9a3c-5cfd153b74c6" containerName="payload-pre-processing"
Sep 15 07:57:44.897779 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:44.897741    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="395d9fa3-23fa-4940-80ea-7c5d77dd73ac" containerName="revoke-keys"
Sep 15 07:57:44.897779 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:44.897747    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="395d9fa3-23fa-4940-80ea-7c5d77dd73ac" containerName="revoke-keys"
Sep 15 07:57:44.897779 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:44.897767    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab4409e7-24c0-465b-b4f0-97481a9d5391" containerName="payload-processing"
Sep 15 07:57:44.897779 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:44.897777    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab4409e7-24c0-465b-b4f0-97481a9d5391" containerName="payload-processing"
Sep 15 07:57:44.898051 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:44.897862    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="395d9fa3-23fa-4940-80ea-7c5d77dd73ac" containerName="revoke-keys"
Sep 15 07:57:44.898051 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:44.897871    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8000dd9-435e-4444-9a3c-5cfd153b74c6" containerName="payload-pre-processing"
Sep 15 07:57:44.898051 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:44.897880    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab4409e7-24c0-465b-b4f0-97481a9d5391" containerName="payload-processing"
Sep 15 07:57:44.901198 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:44.901181    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d58bd-qwvlr"
Sep 15 07:57:44.903780 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:44.903756    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 07:57:44.903917 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:44.903848    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8jpwk\""
Sep 15 07:57:44.907859 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:44.907833    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d58bd-qwvlr"]
Sep 15 07:57:45.005627 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:45.005594    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e527e89-504b-4097-9b2f-3f6223453458-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5d58bd-qwvlr\" (UID: \"9e527e89-504b-4097-9b2f-3f6223453458\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d58bd-qwvlr"
Sep 15 07:57:45.106278 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:45.106216    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e527e89-504b-4097-9b2f-3f6223453458-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5d58bd-qwvlr\" (UID: \"9e527e89-504b-4097-9b2f-3f6223453458\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d58bd-qwvlr"
Sep 15 07:57:45.106822 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:45.106801    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e527e89-504b-4097-9b2f-3f6223453458-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5d58bd-qwvlr\" (UID: \"9e527e89-504b-4097-9b2f-3f6223453458\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d58bd-qwvlr"
Sep 15 07:57:45.211939 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:45.211909    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d58bd-qwvlr"
Sep 15 07:57:45.484952 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:45.484922    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d58bd-qwvlr"]
Sep 15 07:57:45.486956 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:57:45.486923    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e527e89_504b_4097_9b2f_3f6223453458.slice/crio-494f94947bd010398110b89643848174399b9b6c5c88ea509dc634afcd668c71 WatchSource:0}: Error finding container 494f94947bd010398110b89643848174399b9b6c5c88ea509dc634afcd668c71: Status 404 returned error can't find the container with id 494f94947bd010398110b89643848174399b9b6c5c88ea509dc634afcd668c71
Sep 15 07:57:46.015653 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:46.015622    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d58bd-qwvlr" event={"ID":"9e527e89-504b-4097-9b2f-3f6223453458","Type":"ContainerStarted","Data":"9f43c8aabe5a242d95681b10cc04bbe02539bd4feae6c14fb90d6e4cd2a41026"}
Sep 15 07:57:46.016018 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:46.015671    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d58bd-qwvlr" event={"ID":"9e527e89-504b-4097-9b2f-3f6223453458","Type":"ContainerStarted","Data":"494f94947bd010398110b89643848174399b9b6c5c88ea509dc634afcd668c71"}
Sep 15 07:57:46.035084 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:46.035036    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d58bd-qwvlr" podStartSLOduration=2.035020721 podStartE2EDuration="2.035020721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:57:46.034322489 +0000 UTC m=+2370.580892608" watchObservedRunningTime="2026-09-15 07:57:46.035020721 +0000 UTC m=+2370.581590820"
Sep 15 07:57:47.022888 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:47.022846    2571 generic.go:356] "Generic (PLEG): container finished" podID="9e527e89-504b-4097-9b2f-3f6223453458" containerID="9f43c8aabe5a242d95681b10cc04bbe02539bd4feae6c14fb90d6e4cd2a41026" exitCode=0
Sep 15 07:57:47.023297 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:47.022940    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d58bd-qwvlr" event={"ID":"9e527e89-504b-4097-9b2f-3f6223453458","Type":"ContainerDied","Data":"9f43c8aabe5a242d95681b10cc04bbe02539bd4feae6c14fb90d6e4cd2a41026"}
Sep 15 07:57:48.162930 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:48.162906    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d58bd-qwvlr"
Sep 15 07:57:48.336294 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:48.336184    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e527e89-504b-4097-9b2f-3f6223453458-maas-api-service-ca\") pod \"9e527e89-504b-4097-9b2f-3f6223453458\" (UID: \"9e527e89-504b-4097-9b2f-3f6223453458\") "
Sep 15 07:57:48.336621 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:48.336598    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e527e89-504b-4097-9b2f-3f6223453458-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9e527e89-504b-4097-9b2f-3f6223453458" (UID: "9e527e89-504b-4097-9b2f-3f6223453458"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:57:48.437471 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:48.437439    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e527e89-504b-4097-9b2f-3f6223453458-maas-api-service-ca\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:57:49.035265 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:49.035220    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d58bd-qwvlr"
Sep 15 07:57:49.035418 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:49.035263    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d58bd-qwvlr" event={"ID":"9e527e89-504b-4097-9b2f-3f6223453458","Type":"ContainerDied","Data":"494f94947bd010398110b89643848174399b9b6c5c88ea509dc634afcd668c71"}
Sep 15 07:57:49.035418 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:49.035309    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="494f94947bd010398110b89643848174399b9b6c5c88ea509dc634afcd668c71"
Sep 15 07:57:55.439144 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:55.439103    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q"]
Sep 15 07:57:55.439638 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:55.439381    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q" podUID="c2080dfb-4d30-41b7-8fd8-2f2af167bc88" containerName="payload-processing" containerID="cri-o://cc0f5db975474fa40f56775629756d287124c2786e55c77f78193f0a3b8928d4" gracePeriod=30
Sep 15 07:57:55.454279 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:55.454177    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58"]
Sep 15 07:57:55.454668 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:55.454608    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58" podUID="34047e84-9f7a-40e3-bc86-a70880df93b9" containerName="payload-pre-processing" containerID="cri-o://d13c7b7d76577741b501d647345525dad2196835a470eb85be8de54468002d70" gracePeriod=30
Sep 15 07:57:55.619290 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:55.619218    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d58bd-qwvlr"]
Sep 15 07:57:55.621710 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:55.621683    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d58bd-qwvlr"]
Sep 15 07:57:55.727714 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:55.727684    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q"
Sep 15 07:57:55.730590 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:55.730571    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58"
Sep 15 07:57:55.905767 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:55.905731    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qlvw\" (UniqueName: \"kubernetes.io/projected/c2080dfb-4d30-41b7-8fd8-2f2af167bc88-kube-api-access-9qlvw\") pod \"c2080dfb-4d30-41b7-8fd8-2f2af167bc88\" (UID: \"c2080dfb-4d30-41b7-8fd8-2f2af167bc88\") "
Sep 15 07:57:55.905968 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:55.905806    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34047e84-9f7a-40e3-bc86-a70880df93b9-plugins-config-volume\") pod \"34047e84-9f7a-40e3-bc86-a70880df93b9\" (UID: \"34047e84-9f7a-40e3-bc86-a70880df93b9\") "
Sep 15 07:57:55.905968 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:55.905845    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2080dfb-4d30-41b7-8fd8-2f2af167bc88-plugins-config-volume\") pod \"c2080dfb-4d30-41b7-8fd8-2f2af167bc88\" (UID: \"c2080dfb-4d30-41b7-8fd8-2f2af167bc88\") "
Sep 15 07:57:55.905968 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:55.905922    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8skl\" (UniqueName: \"kubernetes.io/projected/34047e84-9f7a-40e3-bc86-a70880df93b9-kube-api-access-f8skl\") pod \"34047e84-9f7a-40e3-bc86-a70880df93b9\" (UID: \"34047e84-9f7a-40e3-bc86-a70880df93b9\") "
Sep 15 07:57:55.906501 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:55.906236    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34047e84-9f7a-40e3-bc86-a70880df93b9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "34047e84-9f7a-40e3-bc86-a70880df93b9" (UID: "34047e84-9f7a-40e3-bc86-a70880df93b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:57:55.906501 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:55.906294    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2080dfb-4d30-41b7-8fd8-2f2af167bc88-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c2080dfb-4d30-41b7-8fd8-2f2af167bc88" (UID: "c2080dfb-4d30-41b7-8fd8-2f2af167bc88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:57:55.908052 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:55.908027    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2080dfb-4d30-41b7-8fd8-2f2af167bc88-kube-api-access-9qlvw" (OuterVolumeSpecName: "kube-api-access-9qlvw") pod "c2080dfb-4d30-41b7-8fd8-2f2af167bc88" (UID: "c2080dfb-4d30-41b7-8fd8-2f2af167bc88"). InnerVolumeSpecName "kube-api-access-9qlvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:57:55.908185 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:55.908093    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34047e84-9f7a-40e3-bc86-a70880df93b9-kube-api-access-f8skl" (OuterVolumeSpecName: "kube-api-access-f8skl") pod "34047e84-9f7a-40e3-bc86-a70880df93b9" (UID: "34047e84-9f7a-40e3-bc86-a70880df93b9"). InnerVolumeSpecName "kube-api-access-f8skl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:57:55.914209 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:55.914181    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e527e89-504b-4097-9b2f-3f6223453458" path="/var/lib/kubelet/pods/9e527e89-504b-4097-9b2f-3f6223453458/volumes"
Sep 15 07:57:56.007318 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:56.007203    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qlvw\" (UniqueName: \"kubernetes.io/projected/c2080dfb-4d30-41b7-8fd8-2f2af167bc88-kube-api-access-9qlvw\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:57:56.007318 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:56.007237    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34047e84-9f7a-40e3-bc86-a70880df93b9-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:57:56.007318 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:56.007268    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2080dfb-4d30-41b7-8fd8-2f2af167bc88-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:57:56.007318 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:56.007280    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8skl\" (UniqueName: \"kubernetes.io/projected/34047e84-9f7a-40e3-bc86-a70880df93b9-kube-api-access-f8skl\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:57:56.074546 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:56.074516    2571 generic.go:356] "Generic (PLEG): container finished" podID="34047e84-9f7a-40e3-bc86-a70880df93b9" containerID="d13c7b7d76577741b501d647345525dad2196835a470eb85be8de54468002d70" exitCode=0
Sep 15 07:57:56.074727 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:56.074560    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58"
Sep 15 07:57:56.074727 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:56.074615    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58" event={"ID":"34047e84-9f7a-40e3-bc86-a70880df93b9","Type":"ContainerDied","Data":"d13c7b7d76577741b501d647345525dad2196835a470eb85be8de54468002d70"}
Sep 15 07:57:56.074727 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:56.074669    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58" event={"ID":"34047e84-9f7a-40e3-bc86-a70880df93b9","Type":"ContainerDied","Data":"84923298798de1a3b0abfd9992090d74f1f28d70700896c0663081a3a115eb94"}
Sep 15 07:57:56.074727 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:56.074695    2571 scope.go:117] "RemoveContainer" containerID="d13c7b7d76577741b501d647345525dad2196835a470eb85be8de54468002d70"
Sep 15 07:57:56.076033 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:56.076014    2571 generic.go:356] "Generic (PLEG): container finished" podID="c2080dfb-4d30-41b7-8fd8-2f2af167bc88" containerID="cc0f5db975474fa40f56775629756d287124c2786e55c77f78193f0a3b8928d4" exitCode=0
Sep 15 07:57:56.076119 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:56.076059    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q"
Sep 15 07:57:56.076224 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:56.076190    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q" event={"ID":"c2080dfb-4d30-41b7-8fd8-2f2af167bc88","Type":"ContainerDied","Data":"cc0f5db975474fa40f56775629756d287124c2786e55c77f78193f0a3b8928d4"}
Sep 15 07:57:56.076369 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:56.076278    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q" event={"ID":"c2080dfb-4d30-41b7-8fd8-2f2af167bc88","Type":"ContainerDied","Data":"3cb40e0534b19e38e6ed573ba3ece26f36b5ad0580a5241bc95f2478d29bb536"}
Sep 15 07:57:56.083613 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:56.083594    2571 scope.go:117] "RemoveContainer" containerID="d13c7b7d76577741b501d647345525dad2196835a470eb85be8de54468002d70"
Sep 15 07:57:56.083895 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:57:56.083849    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d13c7b7d76577741b501d647345525dad2196835a470eb85be8de54468002d70\": container with ID starting with d13c7b7d76577741b501d647345525dad2196835a470eb85be8de54468002d70 not found: ID does not exist" containerID="d13c7b7d76577741b501d647345525dad2196835a470eb85be8de54468002d70"
Sep 15 07:57:56.083895 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:56.083874    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13c7b7d76577741b501d647345525dad2196835a470eb85be8de54468002d70"} err="failed to get container status \"d13c7b7d76577741b501d647345525dad2196835a470eb85be8de54468002d70\": rpc error: code = NotFound desc = could not find container \"d13c7b7d76577741b501d647345525dad2196835a470eb85be8de54468002d70\": container with ID starting with d13c7b7d76577741b501d647345525dad2196835a470eb85be8de54468002d70 not found: ID does not exist"
Sep 15 07:57:56.084036 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:56.083896    2571 scope.go:117] "RemoveContainer" containerID="cc0f5db975474fa40f56775629756d287124c2786e55c77f78193f0a3b8928d4"
Sep 15 07:57:56.091577 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:56.091560    2571 scope.go:117] "RemoveContainer" containerID="cc0f5db975474fa40f56775629756d287124c2786e55c77f78193f0a3b8928d4"
Sep 15 07:57:56.091836 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:57:56.091820    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc0f5db975474fa40f56775629756d287124c2786e55c77f78193f0a3b8928d4\": container with ID starting with cc0f5db975474fa40f56775629756d287124c2786e55c77f78193f0a3b8928d4 not found: ID does not exist" containerID="cc0f5db975474fa40f56775629756d287124c2786e55c77f78193f0a3b8928d4"
Sep 15 07:57:56.091900 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:56.091840    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc0f5db975474fa40f56775629756d287124c2786e55c77f78193f0a3b8928d4"} err="failed to get container status \"cc0f5db975474fa40f56775629756d287124c2786e55c77f78193f0a3b8928d4\": rpc error: code = NotFound desc = could not find container \"cc0f5db975474fa40f56775629756d287124c2786e55c77f78193f0a3b8928d4\": container with ID starting with cc0f5db975474fa40f56775629756d287124c2786e55c77f78193f0a3b8928d4 not found: ID does not exist"
Sep 15 07:57:56.097036 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:56.097013    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58"]
Sep 15 07:57:56.100904 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:56.100883    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d58bd-76cb76697d-dmj58"]
Sep 15 07:57:56.110019 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:56.109994    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q"]
Sep 15 07:57:56.113630 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:56.113609    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d58bd-78f47bdcc9-z4f7q"]
Sep 15 07:57:57.909526 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:57.909492    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34047e84-9f7a-40e3-bc86-a70880df93b9" path="/var/lib/kubelet/pods/34047e84-9f7a-40e3-bc86-a70880df93b9/volumes"
Sep 15 07:57:57.909908 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:57:57.909853    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2080dfb-4d30-41b7-8fd8-2f2af167bc88" path="/var/lib/kubelet/pods/c2080dfb-4d30-41b7-8fd8-2f2af167bc88/volumes"
Sep 15 07:58:02.456105 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:02.456067    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-04464b-cp594"]
Sep 15 07:58:02.456556 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:02.456541    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34047e84-9f7a-40e3-bc86-a70880df93b9" containerName="payload-pre-processing"
Sep 15 07:58:02.456613 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:02.456558    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="34047e84-9f7a-40e3-bc86-a70880df93b9" containerName="payload-pre-processing"
Sep 15 07:58:02.456613 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:02.456568    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2080dfb-4d30-41b7-8fd8-2f2af167bc88" containerName="payload-processing"
Sep 15 07:58:02.456613 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:02.456575    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2080dfb-4d30-41b7-8fd8-2f2af167bc88" containerName="payload-processing"
Sep 15 07:58:02.456613 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:02.456585    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e527e89-504b-4097-9b2f-3f6223453458" containerName="revoke-keys"
Sep 15 07:58:02.456613 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:02.456594    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e527e89-504b-4097-9b2f-3f6223453458" containerName="revoke-keys"
Sep 15 07:58:02.456789 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:02.456675    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2080dfb-4d30-41b7-8fd8-2f2af167bc88" containerName="payload-processing"
Sep 15 07:58:02.456789 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:02.456683    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e527e89-504b-4097-9b2f-3f6223453458" containerName="revoke-keys"
Sep 15 07:58:02.456789 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:02.456695    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="34047e84-9f7a-40e3-bc86-a70880df93b9" containerName="payload-pre-processing"
Sep 15 07:58:02.461620 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:02.461597    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-04464b-cp594"
Sep 15 07:58:02.464358 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:02.464324    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 07:58:02.464579 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:02.464558    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8jpwk\""
Sep 15 07:58:02.466629 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:02.466598    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-04464b-cp594"]
Sep 15 07:58:02.466727 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:02.466667    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6d641f-6521-4a79-995b-c59e4ab0a6e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-04464b-cp594\" (UID: \"2c6d641f-6521-4a79-995b-c59e4ab0a6e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-04464b-cp594"
Sep 15 07:58:02.567786 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:02.567752    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6d641f-6521-4a79-995b-c59e4ab0a6e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-04464b-cp594\" (UID: \"2c6d641f-6521-4a79-995b-c59e4ab0a6e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-04464b-cp594"
Sep 15 07:58:02.568470 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:02.568449    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6d641f-6521-4a79-995b-c59e4ab0a6e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-04464b-cp594\" (UID: \"2c6d641f-6521-4a79-995b-c59e4ab0a6e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-04464b-cp594"
Sep 15 07:58:02.774935 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:02.774831    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-04464b-cp594"
Sep 15 07:58:03.043404 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:03.043376    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-04464b-cp594"]
Sep 15 07:58:03.045611 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:58:03.045579    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6d641f_6521_4a79_995b_c59e4ab0a6e4.slice/crio-e5a1e0c0ba7e9855d16516f088c60657adc20e7cfb45b7e258cd3dc7baccd745 WatchSource:0}: Error finding container e5a1e0c0ba7e9855d16516f088c60657adc20e7cfb45b7e258cd3dc7baccd745: Status 404 returned error can't find the container with id e5a1e0c0ba7e9855d16516f088c60657adc20e7cfb45b7e258cd3dc7baccd745
Sep 15 07:58:03.117567 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:03.117534    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-04464b-cp594" event={"ID":"2c6d641f-6521-4a79-995b-c59e4ab0a6e4","Type":"ContainerStarted","Data":"e5a1e0c0ba7e9855d16516f088c60657adc20e7cfb45b7e258cd3dc7baccd745"}
Sep 15 07:58:04.125087 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:04.125053    2571 generic.go:356] "Generic (PLEG): container finished" podID="2c6d641f-6521-4a79-995b-c59e4ab0a6e4" containerID="a67e8ec6df99e4b71829d67cb04a4aebea4c5917f9d8a51d2120e65ea9b8664c" exitCode=0
Sep 15 07:58:04.125491 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:04.125118    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-04464b-cp594" event={"ID":"2c6d641f-6521-4a79-995b-c59e4ab0a6e4","Type":"ContainerDied","Data":"a67e8ec6df99e4b71829d67cb04a4aebea4c5917f9d8a51d2120e65ea9b8664c"}
Sep 15 07:58:05.260400 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:05.260374    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-04464b-cp594"
Sep 15 07:58:05.295401 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:05.295365    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6d641f-6521-4a79-995b-c59e4ab0a6e4-maas-api-service-ca\") pod \"2c6d641f-6521-4a79-995b-c59e4ab0a6e4\" (UID: \"2c6d641f-6521-4a79-995b-c59e4ab0a6e4\") "
Sep 15 07:58:05.295700 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:05.295678    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c6d641f-6521-4a79-995b-c59e4ab0a6e4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c6d641f-6521-4a79-995b-c59e4ab0a6e4" (UID: "2c6d641f-6521-4a79-995b-c59e4ab0a6e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:58:05.396862 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:05.396777    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6d641f-6521-4a79-995b-c59e4ab0a6e4-maas-api-service-ca\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:58:06.137661 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:06.137624    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-04464b-cp594"
Sep 15 07:58:06.137826 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:06.137624    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-04464b-cp594" event={"ID":"2c6d641f-6521-4a79-995b-c59e4ab0a6e4","Type":"ContainerDied","Data":"e5a1e0c0ba7e9855d16516f088c60657adc20e7cfb45b7e258cd3dc7baccd745"}
Sep 15 07:58:06.137826 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:06.137752    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5a1e0c0ba7e9855d16516f088c60657adc20e7cfb45b7e258cd3dc7baccd745"
Sep 15 07:58:13.179154 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:13.179121    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-04464b-29824305-qxv5t"]
Sep 15 07:58:13.181138 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:13.181116    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-04464b-29824305-qxv5t"]
Sep 15 07:58:13.214967 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:13.214934    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz"]
Sep 15 07:58:13.215246 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:13.215196    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz" podUID="e4f79da6-cffd-4b19-8e53-da4eb7eaca81" containerName="payload-pre-processing" containerID="cri-o://16569fb0034aa0d6f35bbe93679f39c35b25b1475c2c460bb4216a243b9e6912" gracePeriod=30
Sep 15 07:58:13.379359 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:13.379323    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-04464b-cp594"]
Sep 15 07:58:13.381909 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:13.381885    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-04464b-cp594"]
Sep 15 07:58:13.474001 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:13.473976    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz"
Sep 15 07:58:13.571194 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:13.571157    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqs46\" (UniqueName: \"kubernetes.io/projected/e4f79da6-cffd-4b19-8e53-da4eb7eaca81-kube-api-access-dqs46\") pod \"e4f79da6-cffd-4b19-8e53-da4eb7eaca81\" (UID: \"e4f79da6-cffd-4b19-8e53-da4eb7eaca81\") "
Sep 15 07:58:13.571432 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:13.571218    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f79da6-cffd-4b19-8e53-da4eb7eaca81-plugins-config-volume\") pod \"e4f79da6-cffd-4b19-8e53-da4eb7eaca81\" (UID: \"e4f79da6-cffd-4b19-8e53-da4eb7eaca81\") "
Sep 15 07:58:13.571646 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:13.571620    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f79da6-cffd-4b19-8e53-da4eb7eaca81-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4f79da6-cffd-4b19-8e53-da4eb7eaca81" (UID: "e4f79da6-cffd-4b19-8e53-da4eb7eaca81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:58:13.573225 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:13.573204    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f79da6-cffd-4b19-8e53-da4eb7eaca81-kube-api-access-dqs46" (OuterVolumeSpecName: "kube-api-access-dqs46") pod "e4f79da6-cffd-4b19-8e53-da4eb7eaca81" (UID: "e4f79da6-cffd-4b19-8e53-da4eb7eaca81"). InnerVolumeSpecName "kube-api-access-dqs46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:58:13.672695 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:13.672664    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqs46\" (UniqueName: \"kubernetes.io/projected/e4f79da6-cffd-4b19-8e53-da4eb7eaca81-kube-api-access-dqs46\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:58:13.672695 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:13.672692    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f79da6-cffd-4b19-8e53-da4eb7eaca81-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:58:13.909729 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:13.909649    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c6d641f-6521-4a79-995b-c59e4ab0a6e4" path="/var/lib/kubelet/pods/2c6d641f-6521-4a79-995b-c59e4ab0a6e4/volumes"
Sep 15 07:58:13.909955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:13.909941    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8070eeec-fbdd-42d5-8c06-a195b2a25eb4" path="/var/lib/kubelet/pods/8070eeec-fbdd-42d5-8c06-a195b2a25eb4/volumes"
Sep 15 07:58:14.182956 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:14.182861    2571 generic.go:356] "Generic (PLEG): container finished" podID="e4f79da6-cffd-4b19-8e53-da4eb7eaca81" containerID="16569fb0034aa0d6f35bbe93679f39c35b25b1475c2c460bb4216a243b9e6912" exitCode=0
Sep 15 07:58:14.182956 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:14.182909    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz" event={"ID":"e4f79da6-cffd-4b19-8e53-da4eb7eaca81","Type":"ContainerDied","Data":"16569fb0034aa0d6f35bbe93679f39c35b25b1475c2c460bb4216a243b9e6912"}
Sep 15 07:58:14.182956 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:14.182914    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz"
Sep 15 07:58:14.182956 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:14.182952    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz" event={"ID":"e4f79da6-cffd-4b19-8e53-da4eb7eaca81","Type":"ContainerDied","Data":"88e3e38822ed4f3695c274263388ef406b9b9f0e2d3c6ae5d0a2a9dd44c4839f"}
Sep 15 07:58:14.183563 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:14.182974    2571 scope.go:117] "RemoveContainer" containerID="16569fb0034aa0d6f35bbe93679f39c35b25b1475c2c460bb4216a243b9e6912"
Sep 15 07:58:14.191982 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:14.191958    2571 scope.go:117] "RemoveContainer" containerID="16569fb0034aa0d6f35bbe93679f39c35b25b1475c2c460bb4216a243b9e6912"
Sep 15 07:58:14.192249 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:58:14.192222    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16569fb0034aa0d6f35bbe93679f39c35b25b1475c2c460bb4216a243b9e6912\": container with ID starting with 16569fb0034aa0d6f35bbe93679f39c35b25b1475c2c460bb4216a243b9e6912 not found: ID does not exist" containerID="16569fb0034aa0d6f35bbe93679f39c35b25b1475c2c460bb4216a243b9e6912"
Sep 15 07:58:14.192354 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:14.192285    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16569fb0034aa0d6f35bbe93679f39c35b25b1475c2c460bb4216a243b9e6912"} err="failed to get container status \"16569fb0034aa0d6f35bbe93679f39c35b25b1475c2c460bb4216a243b9e6912\": rpc error: code = NotFound desc = could not find container \"16569fb0034aa0d6f35bbe93679f39c35b25b1475c2c460bb4216a243b9e6912\": container with ID starting with 16569fb0034aa0d6f35bbe93679f39c35b25b1475c2c460bb4216a243b9e6912 not found: ID does not exist"
Sep 15 07:58:14.200957 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:14.200930    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz"]
Sep 15 07:58:14.204634 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:14.204611    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-04464b-57979cc8c4-bdmjz"]
Sep 15 07:58:15.908690 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:15.908657    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4f79da6-cffd-4b19-8e53-da4eb7eaca81" path="/var/lib/kubelet/pods/e4f79da6-cffd-4b19-8e53-da4eb7eaca81/volumes"
Sep 15 07:58:16.344851 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:16.344813    2571 scope.go:117] "RemoveContainer" containerID="42809aa23832d8b3ac3fc69fcd3c3144de016ca93de8d8f6cd7ec71c65c6407e"
Sep 15 07:58:20.141977 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:20.141940    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e3a482-cnzn5"]
Sep 15 07:58:20.142397 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:20.142352    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4f79da6-cffd-4b19-8e53-da4eb7eaca81" containerName="payload-pre-processing"
Sep 15 07:58:20.142397 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:20.142364    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4f79da6-cffd-4b19-8e53-da4eb7eaca81" containerName="payload-pre-processing"
Sep 15 07:58:20.142397 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:20.142380    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c6d641f-6521-4a79-995b-c59e4ab0a6e4" containerName="revoke-keys"
Sep 15 07:58:20.142397 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:20.142387    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c6d641f-6521-4a79-995b-c59e4ab0a6e4" containerName="revoke-keys"
Sep 15 07:58:20.142613 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:20.142458    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4f79da6-cffd-4b19-8e53-da4eb7eaca81" containerName="payload-pre-processing"
Sep 15 07:58:20.142613 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:20.142468    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c6d641f-6521-4a79-995b-c59e4ab0a6e4" containerName="revoke-keys"
Sep 15 07:58:20.146861 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:20.146842    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e3a482-cnzn5"
Sep 15 07:58:20.149238 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:20.149215    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 07:58:20.149347 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:20.149277    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8jpwk\""
Sep 15 07:58:20.152657 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:20.152620    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e3a482-cnzn5"]
Sep 15 07:58:20.234397 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:20.234366    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/822ddd9f-b5cf-4ed7-a48d-16a30ecf7236-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e3a482-cnzn5\" (UID: \"822ddd9f-b5cf-4ed7-a48d-16a30ecf7236\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e3a482-cnzn5"
Sep 15 07:58:20.336877 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:20.336832    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/822ddd9f-b5cf-4ed7-a48d-16a30ecf7236-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e3a482-cnzn5\" (UID: \"822ddd9f-b5cf-4ed7-a48d-16a30ecf7236\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e3a482-cnzn5"
Sep 15 07:58:20.337653 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:20.337629    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/822ddd9f-b5cf-4ed7-a48d-16a30ecf7236-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e3a482-cnzn5\" (UID: \"822ddd9f-b5cf-4ed7-a48d-16a30ecf7236\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e3a482-cnzn5"
Sep 15 07:58:20.458643 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:20.458613    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e3a482-cnzn5"
Sep 15 07:58:20.727112 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:20.727084    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e3a482-cnzn5"]
Sep 15 07:58:20.729497 ip-10-0-129-76 kubenswrapper[2571]: W0915 07:58:20.729467    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822ddd9f_b5cf_4ed7_a48d_16a30ecf7236.slice/crio-2352553d1a3bf4ec149e35fc57928b4f299fb34de7a4a29f4d22cd660013bbd1 WatchSource:0}: Error finding container 2352553d1a3bf4ec149e35fc57928b4f299fb34de7a4a29f4d22cd660013bbd1: Status 404 returned error can't find the container with id 2352553d1a3bf4ec149e35fc57928b4f299fb34de7a4a29f4d22cd660013bbd1
Sep 15 07:58:21.221541 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:21.221501    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e3a482-cnzn5" event={"ID":"822ddd9f-b5cf-4ed7-a48d-16a30ecf7236","Type":"ContainerStarted","Data":"b6112304a9fd824f20593941cbd64f54ca3682f2184d613afcb1692d48459da2"}
Sep 15 07:58:21.221917 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:21.221565    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e3a482-cnzn5" event={"ID":"822ddd9f-b5cf-4ed7-a48d-16a30ecf7236","Type":"ContainerStarted","Data":"2352553d1a3bf4ec149e35fc57928b4f299fb34de7a4a29f4d22cd660013bbd1"}
Sep 15 07:58:21.238491 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:21.238388    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e3a482-cnzn5" podStartSLOduration=1.238372836 podStartE2EDuration="1.238372836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:58:21.23595453 +0000 UTC m=+2405.782524643" watchObservedRunningTime="2026-09-15 07:58:21.238372836 +0000 UTC m=+2405.784942956"
Sep 15 07:58:22.231327 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:22.231291    2571 generic.go:356] "Generic (PLEG): container finished" podID="822ddd9f-b5cf-4ed7-a48d-16a30ecf7236" containerID="b6112304a9fd824f20593941cbd64f54ca3682f2184d613afcb1692d48459da2" exitCode=0
Sep 15 07:58:22.231711 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:22.231385    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e3a482-cnzn5" event={"ID":"822ddd9f-b5cf-4ed7-a48d-16a30ecf7236","Type":"ContainerDied","Data":"b6112304a9fd824f20593941cbd64f54ca3682f2184d613afcb1692d48459da2"}
Sep 15 07:58:23.365419 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:23.365393    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e3a482-cnzn5"
Sep 15 07:58:23.469347 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:23.469316    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/822ddd9f-b5cf-4ed7-a48d-16a30ecf7236-maas-api-service-ca\") pod \"822ddd9f-b5cf-4ed7-a48d-16a30ecf7236\" (UID: \"822ddd9f-b5cf-4ed7-a48d-16a30ecf7236\") "
Sep 15 07:58:23.469666 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:23.469643    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/822ddd9f-b5cf-4ed7-a48d-16a30ecf7236-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "822ddd9f-b5cf-4ed7-a48d-16a30ecf7236" (UID: "822ddd9f-b5cf-4ed7-a48d-16a30ecf7236"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:58:23.570211 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:23.570128    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/822ddd9f-b5cf-4ed7-a48d-16a30ecf7236-maas-api-service-ca\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:58:24.244204 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:24.244171    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e3a482-cnzn5"
Sep 15 07:58:24.244425 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:24.244171    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e3a482-cnzn5" event={"ID":"822ddd9f-b5cf-4ed7-a48d-16a30ecf7236","Type":"ContainerDied","Data":"2352553d1a3bf4ec149e35fc57928b4f299fb34de7a4a29f4d22cd660013bbd1"}
Sep 15 07:58:24.244425 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:24.244336    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2352553d1a3bf4ec149e35fc57928b4f299fb34de7a4a29f4d22cd660013bbd1"
Sep 15 07:58:30.789076 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:30.789036    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p"]
Sep 15 07:58:30.789560 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:30.789311    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p" podUID="45fcb589-c2be-4abf-b417-293704470729" containerName="payload-pre-processing" containerID="cri-o://cec86f353ddde39d25b652f4f5a11cd15d0c16980c8d285760c03f8a94d5a54a" gracePeriod=30
Sep 15 07:58:30.938377 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:30.938338    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e3a482-cnzn5"]
Sep 15 07:58:30.940186 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:30.940154    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e3a482-cnzn5"]
Sep 15 07:58:31.046005 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:31.045944    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p"
Sep 15 07:58:31.141918 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:31.141880    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t56q5\" (UniqueName: \"kubernetes.io/projected/45fcb589-c2be-4abf-b417-293704470729-kube-api-access-t56q5\") pod \"45fcb589-c2be-4abf-b417-293704470729\" (UID: \"45fcb589-c2be-4abf-b417-293704470729\") "
Sep 15 07:58:31.142089 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:31.141929    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45fcb589-c2be-4abf-b417-293704470729-plugins-config-volume\") pod \"45fcb589-c2be-4abf-b417-293704470729\" (UID: \"45fcb589-c2be-4abf-b417-293704470729\") "
Sep 15 07:58:31.142379 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:31.142355    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45fcb589-c2be-4abf-b417-293704470729-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "45fcb589-c2be-4abf-b417-293704470729" (UID: "45fcb589-c2be-4abf-b417-293704470729"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:58:31.143973 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:31.143950    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45fcb589-c2be-4abf-b417-293704470729-kube-api-access-t56q5" (OuterVolumeSpecName: "kube-api-access-t56q5") pod "45fcb589-c2be-4abf-b417-293704470729" (UID: "45fcb589-c2be-4abf-b417-293704470729"). InnerVolumeSpecName "kube-api-access-t56q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:58:31.243079 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:31.243047    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t56q5\" (UniqueName: \"kubernetes.io/projected/45fcb589-c2be-4abf-b417-293704470729-kube-api-access-t56q5\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:58:31.243079 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:31.243072    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45fcb589-c2be-4abf-b417-293704470729-plugins-config-volume\") on node \"ip-10-0-129-76.ec2.internal\" DevicePath \"\""
Sep 15 07:58:31.283128 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:31.283096    2571 generic.go:356] "Generic (PLEG): container finished" podID="45fcb589-c2be-4abf-b417-293704470729" containerID="cec86f353ddde39d25b652f4f5a11cd15d0c16980c8d285760c03f8a94d5a54a" exitCode=0
Sep 15 07:58:31.283340 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:31.283149    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p"
Sep 15 07:58:31.283340 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:31.283187    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p" event={"ID":"45fcb589-c2be-4abf-b417-293704470729","Type":"ContainerDied","Data":"cec86f353ddde39d25b652f4f5a11cd15d0c16980c8d285760c03f8a94d5a54a"}
Sep 15 07:58:31.283340 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:31.283238    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p" event={"ID":"45fcb589-c2be-4abf-b417-293704470729","Type":"ContainerDied","Data":"dc951d2ce40557f6a415a2a49cc6f45d0d5621ec0b2622a687601cef26cf839e"}
Sep 15 07:58:31.283340 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:31.283273    2571 scope.go:117] "RemoveContainer" containerID="cec86f353ddde39d25b652f4f5a11cd15d0c16980c8d285760c03f8a94d5a54a"
Sep 15 07:58:31.292955 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:31.292938    2571 scope.go:117] "RemoveContainer" containerID="cec86f353ddde39d25b652f4f5a11cd15d0c16980c8d285760c03f8a94d5a54a"
Sep 15 07:58:31.293319 ip-10-0-129-76 kubenswrapper[2571]: E0915 07:58:31.293297    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cec86f353ddde39d25b652f4f5a11cd15d0c16980c8d285760c03f8a94d5a54a\": container with ID starting with cec86f353ddde39d25b652f4f5a11cd15d0c16980c8d285760c03f8a94d5a54a not found: ID does not exist" containerID="cec86f353ddde39d25b652f4f5a11cd15d0c16980c8d285760c03f8a94d5a54a"
Sep 15 07:58:31.293381 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:31.293327    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec86f353ddde39d25b652f4f5a11cd15d0c16980c8d285760c03f8a94d5a54a"} err="failed to get container status \"cec86f353ddde39d25b652f4f5a11cd15d0c16980c8d285760c03f8a94d5a54a\": rpc error: code = NotFound desc = could not find container \"cec86f353ddde39d25b652f4f5a11cd15d0c16980c8d285760c03f8a94d5a54a\": container with ID starting with cec86f353ddde39d25b652f4f5a11cd15d0c16980c8d285760c03f8a94d5a54a not found: ID does not exist"
Sep 15 07:58:31.304954 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:31.304901    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p"]
Sep 15 07:58:31.308461 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:31.308439    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e3a482-6d5877857d-zwz5p"]
Sep 15 07:58:31.909412 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:31.909371    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45fcb589-c2be-4abf-b417-293704470729" path="/var/lib/kubelet/pods/45fcb589-c2be-4abf-b417-293704470729/volumes"
Sep 15 07:58:31.909896 ip-10-0-129-76 kubenswrapper[2571]: I0915 07:58:31.909877    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="822ddd9f-b5cf-4ed7-a48d-16a30ecf7236" path="/var/lib/kubelet/pods/822ddd9f-b5cf-4ed7-a48d-16a30ecf7236/volumes"
Sep 15 08:00:00.136817 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:00:00.136777    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824320-v6pmr"]
Sep 15 08:00:00.139282 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:00:00.137189    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="822ddd9f-b5cf-4ed7-a48d-16a30ecf7236" containerName="revoke-keys"
Sep 15 08:00:00.139282 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:00:00.137201    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="822ddd9f-b5cf-4ed7-a48d-16a30ecf7236" containerName="revoke-keys"
Sep 15 08:00:00.139282 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:00:00.137219    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45fcb589-c2be-4abf-b417-293704470729" containerName="payload-pre-processing"
Sep 15 08:00:00.139282 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:00:00.137225    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="45fcb589-c2be-4abf-b417-293704470729" containerName="payload-pre-processing"
Sep 15 08:00:00.139282 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:00:00.137303    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="45fcb589-c2be-4abf-b417-293704470729" containerName="payload-pre-processing"
Sep 15 08:00:00.139282 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:00:00.137312    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="822ddd9f-b5cf-4ed7-a48d-16a30ecf7236" containerName="revoke-keys"
Sep 15 08:00:00.140269 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:00:00.140239    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824320-v6pmr"
Sep 15 08:00:00.142465 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:00:00.142442    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8jpwk\""
Sep 15 08:00:00.146573 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:00:00.146548    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824320-v6pmr"]
Sep 15 08:00:00.150957 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:00:00.150936    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824320-v6pmr"
Sep 15 08:00:00.417377 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:00:00.417352    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824320-v6pmr"]
Sep 15 08:00:00.419346 ip-10-0-129-76 kubenswrapper[2571]: W0915 08:00:00.419316    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46845c11_74f3_42e3_8be4_61ab50ddb852.slice/crio-c56f3230b50ea040b6c3c52c98cc93ad2589e632e9b8770bd8cbcfde32f3ab7b WatchSource:0}: Error finding container c56f3230b50ea040b6c3c52c98cc93ad2589e632e9b8770bd8cbcfde32f3ab7b: Status 404 returned error can't find the container with id c56f3230b50ea040b6c3c52c98cc93ad2589e632e9b8770bd8cbcfde32f3ab7b
Sep 15 08:00:00.745868 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:00:00.745822    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824320-v6pmr" event={"ID":"46845c11-74f3-42e3-8be4-61ab50ddb852","Type":"ContainerStarted","Data":"143fc8a52e5b0c5927df2bd3894b7331fc0f743fc69e821a09659d2e708545be"}
Sep 15 08:00:00.746056 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:00:00.745886    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824320-v6pmr" event={"ID":"46845c11-74f3-42e3-8be4-61ab50ddb852","Type":"ContainerStarted","Data":"c56f3230b50ea040b6c3c52c98cc93ad2589e632e9b8770bd8cbcfde32f3ab7b"}
Sep 15 08:00:00.761508 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:00:00.761458    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824320-v6pmr" podStartSLOduration=0.76144163 podStartE2EDuration="761.44163ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 08:00:00.759490694 +0000 UTC m=+2505.306060817" watchObservedRunningTime="2026-09-15 08:00:00.76144163 +0000 UTC m=+2505.308011729"
Sep 15 08:00:01.752308 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:00:01.752273    2571 generic.go:356] "Generic (PLEG): container finished" podID="46845c11-74f3-42e3-8be4-61ab50ddb852" containerID="143fc8a52e5b0c5927df2bd3894b7331fc0f743fc69e821a09659d2e708545be" exitCode=0
Sep 15 08:00:01.752696 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:00:01.752342    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824320-v6pmr" event={"ID":"46845c11-74f3-42e3-8be4-61ab50ddb852","Type":"ContainerDied","Data":"143fc8a52e5b0c5927df2bd3894b7331fc0f743fc69e821a09659d2e708545be"}
Sep 15 08:00:02.917320 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:00:02.917288    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824320-v6pmr"
Sep 15 08:00:03.765166 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:00:03.765115    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824320-v6pmr" event={"ID":"46845c11-74f3-42e3-8be4-61ab50ddb852","Type":"ContainerDied","Data":"c56f3230b50ea040b6c3c52c98cc93ad2589e632e9b8770bd8cbcfde32f3ab7b"}
Sep 15 08:00:03.765364 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:00:03.765205    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c56f3230b50ea040b6c3c52c98cc93ad2589e632e9b8770bd8cbcfde32f3ab7b"
Sep 15 08:00:03.765364 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:00:03.765145    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824320-v6pmr"
Sep 15 08:03:16.411500 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:03:16.411422    2571 scope.go:117] "RemoveContainer" containerID="76b190fbefb1cd121d39910b70ab43396c0a55bda9a8591409d9f57ef5e85e10"
Sep 15 08:03:16.420173 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:03:16.420150    2571 scope.go:117] "RemoveContainer" containerID="2274ce7a25bb299fe196261ed50b09dfedde596ebc678b65850b892495b15190"
Sep 15 08:04:16.447138 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:04:16.447100    2571 scope.go:117] "RemoveContainer" containerID="9f43c8aabe5a242d95681b10cc04bbe02539bd4feae6c14fb90d6e4cd2a41026"
Sep 15 08:04:16.461915 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:04:16.461890    2571 scope.go:117] "RemoveContainer" containerID="a67e8ec6df99e4b71829d67cb04a4aebea4c5917f9d8a51d2120e65ea9b8664c"
Sep 15 08:04:45.542075 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:04:45.542046    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565d545f7b-7djq8_c74ca08a-5740-4e74-b9b1-611d28dc143d/copy-manifests/0.log"
Sep 15 08:04:45.548716 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:04:45.548689    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565d545f7b-7djq8_c74ca08a-5740-4e74-b9b1-611d28dc143d/manager/0.log"
Sep 15 08:04:46.377021 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:04:46.376991    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29824305-jjpqg_bc9dd07f-838f-4c4e-9167-cb4cc46e1731/cleanup/0.log"
Sep 15 08:04:46.495215 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:04:46.495188    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29824320-v6pmr_46845c11-74f3-42e3-8be4-61ab50ddb852/cleanup/0.log"
Sep 15 08:04:47.819503 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:04:47.819475    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-6gp2q_1012ba59-824f-42bb-bd39-3c2ebac8e491/kuadrant-console-plugin/0.log"
Sep 15 08:04:48.195094 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:04:48.195009    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-2zv8v_21d46833-6a60-46fb-8498-87a141821243/manager/0.log"
Sep 15 08:04:48.779278 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:04:48.779228    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77b7bc76bc-mrxgc_02c7ea90-3665-4228-9470-871cebea1479/kube-auth-proxy/0.log"
Sep 15 08:04:49.125222 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:04:49.125141    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-ghksm_9c9b59cc-5710-45db-8109-8efa36ab8e3d/payload-pre-processing/0.log"
Sep 15 08:04:49.899820 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:04:49.899791    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-ml26w_a84a09c4-85e1-4440-acf7-ec9c93e9514f/main/0.log"
Sep 15 08:04:50.132190 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:04:50.132155    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-bbcpk_8fcb590e-287c-4292-afc9-16fcbb5c12f8/main/0.log"
Sep 15 08:04:50.239173 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:04:50.239142    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4l2q4_c04ccc49-3dfe-49a3-9435-0696d963afce/main/0.log"
Sep 15 08:04:50.494043 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:04:50.493966    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-dvx5v_a80b374e-6518-4173-89a8-097cea66070f/main/0.log"
Sep 15 08:04:58.592094 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:04:58.592060    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kzskh_08733b0b-6ce9-4bc5-87c5-4565bf0833b5/global-pull-secret-syncer/0.log"
Sep 15 08:04:58.708726 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:04:58.708691    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dclwl_061e1b77-3f2d-4f1a-b389-04ffdf5d0530/konnectivity-agent/0.log"
Sep 15 08:04:58.730182 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:04:58.730156    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-76.ec2.internal_457fc68f7214b4c70c98b4a1d69ed2ba/haproxy/0.log"
Sep 15 08:05:04.010730 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:04.010645    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-6gp2q_1012ba59-824f-42bb-bd39-3c2ebac8e491/kuadrant-console-plugin/0.log"
Sep 15 08:05:04.168347 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:04.168312    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-2zv8v_21d46833-6a60-46fb-8498-87a141821243/manager/0.log"
Sep 15 08:05:05.677876 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:05.677836    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-v6vxp_f86f2096-0758-472d-a436-90432eb7368c/cluster-monitoring-operator/0.log"
Sep 15 08:05:05.711279 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:05.711238    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-jdzrb_0dbc5764-bcfa-4bfd-8e85-dc6430bb3064/kube-state-metrics/0.log"
Sep 15 08:05:05.749862 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:05.749827    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-jdzrb_0dbc5764-bcfa-4bfd-8e85-dc6430bb3064/kube-rbac-proxy-main/0.log"
Sep 15 08:05:05.782954 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:05.782921    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-jdzrb_0dbc5764-bcfa-4bfd-8e85-dc6430bb3064/kube-rbac-proxy-self/0.log"
Sep 15 08:05:05.817087 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:05.817057    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-8b6949c-j2xf4_0b3c282e-4fc2-4c16-bfa5-cd7ba72a02d9/metrics-server/0.log"
Sep 15 08:05:05.862579 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:05.862544    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-chnj2_d8499382-718b-41c0-a32e-ca7118b6efd0/monitoring-plugin/0.log"
Sep 15 08:05:06.087981 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:06.087954    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wr8bg_7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc/node-exporter/0.log"
Sep 15 08:05:06.108120 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:06.108087    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wr8bg_7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc/kube-rbac-proxy/0.log"
Sep 15 08:05:06.129074 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:06.129046    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wr8bg_7f6cb60b-f7af-49bb-8a3e-70f1ce0c0dfc/init-textfile/0.log"
Sep 15 08:05:06.155666 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:06.155633    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-b7cpz_ad63d264-6111-4b9c-be64-99e35ef4c05a/kube-rbac-proxy-main/0.log"
Sep 15 08:05:06.178486 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:06.178456    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-b7cpz_ad63d264-6111-4b9c-be64-99e35ef4c05a/kube-rbac-proxy-self/0.log"
Sep 15 08:05:06.201357 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:06.201326    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-b7cpz_ad63d264-6111-4b9c-be64-99e35ef4c05a/openshift-state-metrics/0.log"
Sep 15 08:05:06.247119 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:06.247087    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b/prometheus/0.log"
Sep 15 08:05:06.278773 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:06.278744    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b/config-reloader/0.log"
Sep 15 08:05:06.304196 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:06.304167    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b/thanos-sidecar/0.log"
Sep 15 08:05:06.326742 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:06.326712    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b/kube-rbac-proxy-web/0.log"
Sep 15 08:05:06.349776 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:06.349692    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b/kube-rbac-proxy/0.log"
Sep 15 08:05:06.370882 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:06.370850    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b/kube-rbac-proxy-thanos/0.log"
Sep 15 08:05:06.393590 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:06.393558    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4bf75cf-6cca-40b5-ae06-d3e6a9e3993b/init-config-reloader/0.log"
Sep 15 08:05:06.575785 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:06.575744    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67fdb78968-hf97j_4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62/thanos-query/0.log"
Sep 15 08:05:06.596552 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:06.596523    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67fdb78968-hf97j_4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62/kube-rbac-proxy-web/0.log"
Sep 15 08:05:06.616526 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:06.616431    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67fdb78968-hf97j_4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62/kube-rbac-proxy/0.log"
Sep 15 08:05:06.638573 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:06.638544    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67fdb78968-hf97j_4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62/prom-label-proxy/0.log"
Sep 15 08:05:06.661773 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:06.661742    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67fdb78968-hf97j_4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62/kube-rbac-proxy-rules/0.log"
Sep 15 08:05:06.685231 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:06.685202    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67fdb78968-hf97j_4b2ec3c6-7c55-4e3e-bd10-ca26b10bfc62/kube-rbac-proxy-metrics/0.log"
Sep 15 08:05:07.918495 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:07.918466    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-wdl2x_6a8acf07-ffd3-4101-a1d5-16770855954c/networking-console-plugin/0.log"
Sep 15 08:05:08.239123 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.239087    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9"]
Sep 15 08:05:08.239494 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.239482    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46845c11-74f3-42e3-8be4-61ab50ddb852" containerName="cleanup"
Sep 15 08:05:08.239546 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.239495    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="46845c11-74f3-42e3-8be4-61ab50ddb852" containerName="cleanup"
Sep 15 08:05:08.239588 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.239578    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="46845c11-74f3-42e3-8be4-61ab50ddb852" containerName="cleanup"
Sep 15 08:05:08.242818 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.242796    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9"
Sep 15 08:05:08.245078 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.245056    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c8zpm\"/\"openshift-service-ca.crt\""
Sep 15 08:05:08.245078 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.245072    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c8zpm\"/\"kube-root-ca.crt\""
Sep 15 08:05:08.245911 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.245887    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c8zpm\"/\"default-dockercfg-5fl5x\""
Sep 15 08:05:08.254124 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.254093    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9"]
Sep 15 08:05:08.379158 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.379119    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1d0959f7-db57-4533-82a3-d351f019e1f4-proc\") pod \"perf-node-gather-daemonset-xw9q9\" (UID: \"1d0959f7-db57-4533-82a3-d351f019e1f4\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9"
Sep 15 08:05:08.379372 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.379167    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d0959f7-db57-4533-82a3-d351f019e1f4-sys\") pod \"perf-node-gather-daemonset-xw9q9\" (UID: \"1d0959f7-db57-4533-82a3-d351f019e1f4\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9"
Sep 15 08:05:08.379372 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.379206    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plw2p\" (UniqueName: \"kubernetes.io/projected/1d0959f7-db57-4533-82a3-d351f019e1f4-kube-api-access-plw2p\") pod \"perf-node-gather-daemonset-xw9q9\" (UID: \"1d0959f7-db57-4533-82a3-d351f019e1f4\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9"
Sep 15 08:05:08.379372 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.379314    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1d0959f7-db57-4533-82a3-d351f019e1f4-podres\") pod \"perf-node-gather-daemonset-xw9q9\" (UID: \"1d0959f7-db57-4533-82a3-d351f019e1f4\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9"
Sep 15 08:05:08.379372 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.379350    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d0959f7-db57-4533-82a3-d351f019e1f4-lib-modules\") pod \"perf-node-gather-daemonset-xw9q9\" (UID: \"1d0959f7-db57-4533-82a3-d351f019e1f4\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9"
Sep 15 08:05:08.480195 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.480148    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1d0959f7-db57-4533-82a3-d351f019e1f4-proc\") pod \"perf-node-gather-daemonset-xw9q9\" (UID: \"1d0959f7-db57-4533-82a3-d351f019e1f4\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9"
Sep 15 08:05:08.480195 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.480196    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d0959f7-db57-4533-82a3-d351f019e1f4-sys\") pod \"perf-node-gather-daemonset-xw9q9\" (UID: \"1d0959f7-db57-4533-82a3-d351f019e1f4\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9"
Sep 15 08:05:08.480475 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.480298    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d0959f7-db57-4533-82a3-d351f019e1f4-sys\") pod \"perf-node-gather-daemonset-xw9q9\" (UID: \"1d0959f7-db57-4533-82a3-d351f019e1f4\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9"
Sep 15 08:05:08.480475 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.480302    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1d0959f7-db57-4533-82a3-d351f019e1f4-proc\") pod \"perf-node-gather-daemonset-xw9q9\" (UID: \"1d0959f7-db57-4533-82a3-d351f019e1f4\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9"
Sep 15 08:05:08.480475 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.480324    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plw2p\" (UniqueName: \"kubernetes.io/projected/1d0959f7-db57-4533-82a3-d351f019e1f4-kube-api-access-plw2p\") pod \"perf-node-gather-daemonset-xw9q9\" (UID: \"1d0959f7-db57-4533-82a3-d351f019e1f4\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9"
Sep 15 08:05:08.480475 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.480424    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1d0959f7-db57-4533-82a3-d351f019e1f4-podres\") pod \"perf-node-gather-daemonset-xw9q9\" (UID: \"1d0959f7-db57-4533-82a3-d351f019e1f4\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9"
Sep 15 08:05:08.480475 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.480449    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d0959f7-db57-4533-82a3-d351f019e1f4-lib-modules\") pod \"perf-node-gather-daemonset-xw9q9\" (UID: \"1d0959f7-db57-4533-82a3-d351f019e1f4\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9"
Sep 15 08:05:08.480725 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.480563    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1d0959f7-db57-4533-82a3-d351f019e1f4-podres\") pod \"perf-node-gather-daemonset-xw9q9\" (UID: \"1d0959f7-db57-4533-82a3-d351f019e1f4\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9"
Sep 15 08:05:08.480725 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.480570    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d0959f7-db57-4533-82a3-d351f019e1f4-lib-modules\") pod \"perf-node-gather-daemonset-xw9q9\" (UID: \"1d0959f7-db57-4533-82a3-d351f019e1f4\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9"
Sep 15 08:05:08.490111 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.490049    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plw2p\" (UniqueName: \"kubernetes.io/projected/1d0959f7-db57-4533-82a3-d351f019e1f4-kube-api-access-plw2p\") pod \"perf-node-gather-daemonset-xw9q9\" (UID: \"1d0959f7-db57-4533-82a3-d351f019e1f4\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9"
Sep 15 08:05:08.556974 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.556935    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9"
Sep 15 08:05:08.825383 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:08.825357    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9"]
Sep 15 08:05:08.828169 ip-10-0-129-76 kubenswrapper[2571]: W0915 08:05:08.828134    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1d0959f7_db57_4533_82a3_d351f019e1f4.slice/crio-a97aaeac84fa341b58ab6d79d70ec14a12028c34a37d0bc5100c06f8166b7f59 WatchSource:0}: Error finding container a97aaeac84fa341b58ab6d79d70ec14a12028c34a37d0bc5100c06f8166b7f59: Status 404 returned error can't find the container with id a97aaeac84fa341b58ab6d79d70ec14a12028c34a37d0bc5100c06f8166b7f59
Sep 15 08:05:09.370852 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:09.370816    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9" event={"ID":"1d0959f7-db57-4533-82a3-d351f019e1f4","Type":"ContainerStarted","Data":"3948f54c033ac290468d3d529ca73c92c2e97421781ce5cceaa9d3d0eeee2bc5"}
Sep 15 08:05:09.371266 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:09.370871    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9" event={"ID":"1d0959f7-db57-4533-82a3-d351f019e1f4","Type":"ContainerStarted","Data":"a97aaeac84fa341b58ab6d79d70ec14a12028c34a37d0bc5100c06f8166b7f59"}
Sep 15 08:05:09.371266 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:09.370943    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9"
Sep 15 08:05:09.386580 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:09.386527    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9" podStartSLOduration=1.386510742 podStartE2EDuration="1.386510742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 08:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 08:05:09.385229434 +0000 UTC m=+2813.931799590" watchObservedRunningTime="2026-09-15 08:05:09.386510742 +0000 UTC m=+2813.933080842"
Sep 15 08:05:09.863891 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:09.863853    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-7kvwg_1891cbd6-62bb-4387-8822-552d9047b120/volume-data-source-validator/0.log"
Sep 15 08:05:10.976592 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:10.976562    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qvk9j_64a7e365-d705-4174-992f-bd271f4b418e/dns/0.log"
Sep 15 08:05:10.998077 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:10.998047    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qvk9j_64a7e365-d705-4174-992f-bd271f4b418e/kube-rbac-proxy/0.log"
Sep 15 08:05:11.067802 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:11.067768    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zvc79_a1932db5-d903-4b17-ac25-36613aa04ba4/dns-node-resolver/0.log"
Sep 15 08:05:11.601669 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:11.601638    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4scf4_c38076ad-5b5b-4fff-8ad9-149153f9d9d2/node-ca/0.log"
Sep 15 08:05:13.429290 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:13.429244    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77b7bc76bc-mrxgc_02c7ea90-3665-4228-9470-871cebea1479/kube-auth-proxy/0.log"
Sep 15 08:05:13.509798 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:13.509757    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-ghksm_9c9b59cc-5710-45db-8109-8efa36ab8e3d/payload-pre-processing/0.log"
Sep 15 08:05:14.449652 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:14.449623    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5ltrc_1934978a-de53-4efa-8cb8-2f2bc312ceaf/serve-healthcheck-canary/0.log"
Sep 15 08:05:15.130791 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:15.130761    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qk9cv_4adb8650-815d-427e-bf5e-f0fb7d41db60/kube-rbac-proxy/0.log"
Sep 15 08:05:15.156501 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:15.156472    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qk9cv_4adb8650-815d-427e-bf5e-f0fb7d41db60/exporter/0.log"
Sep 15 08:05:15.180502 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:15.180474    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qk9cv_4adb8650-815d-427e-bf5e-f0fb7d41db60/extractor/0.log"
Sep 15 08:05:15.385596 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:15.385519    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-xw9q9"
Sep 15 08:05:16.484932 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:16.484890    2571 scope.go:117] "RemoveContainer" containerID="b6112304a9fd824f20593941cbd64f54ca3682f2184d613afcb1692d48459da2"
Sep 15 08:05:18.047182 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:18.047137    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565d545f7b-7djq8_c74ca08a-5740-4e74-b9b1-611d28dc143d/manager/0.log"
Sep 15 08:05:18.068910 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:18.068880    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565d545f7b-7djq8_c74ca08a-5740-4e74-b9b1-611d28dc143d/copy-manifests/0.log"
Sep 15 08:05:19.424732 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:19.424651    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7b9fc96c88-6m7jq_23099786-e37f-4df6-8d3e-38105d2a3f61/manager/0.log"
Sep 15 08:05:19.530815 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:19.530778    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77b7bc76bc-mrxgc_02c7ea90-3665-4228-9470-871cebea1479/kube-auth-proxy/0.log"
Sep 15 08:05:19.593166 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:19.593137    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-ghksm_9c9b59cc-5710-45db-8109-8efa36ab8e3d/payload-pre-processing/0.log"
Sep 15 08:05:21.902053 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:21.902023    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77b7bc76bc-mrxgc_02c7ea90-3665-4228-9470-871cebea1479/kube-auth-proxy/0.log"
Sep 15 08:05:21.963170 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:21.963139    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-ghksm_9c9b59cc-5710-45db-8109-8efa36ab8e3d/payload-pre-processing/0.log"
Sep 15 08:05:24.472398 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:24.472367    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-5rzzs_299e0037-3343-48d4-bb94-080df4472b59/migrator/0.log"
Sep 15 08:05:24.495930 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:24.495899    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-5rzzs_299e0037-3343-48d4-bb94-080df4472b59/graceful-termination/0.log"
Sep 15 08:05:24.844692 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:24.844653    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-j7dlf_ea4c7c35-ce08-453e-b629-f52c0ab26c66/kube-storage-version-migrator-operator/1.log"
Sep 15 08:05:24.845516 ip-10-0-129-76 kubenswrapper[2571]: I0915 08:05:24.845495    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-j7dlf_ea4c7c35-ce08-453e-b629-f52c0ab26c66/kube-storage-version-migrator-operator/0.log"