Sep 10 04:41:01.350151 ip-10-0-132-70 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 04:41:01.779150 ip-10-0-132-70 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 04:41:01.779150 ip-10-0-132-70 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 04:41:01.779150 ip-10-0-132-70 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 04:41:01.779150 ip-10-0-132-70 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 04:41:01.779150 ip-10-0-132-70 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 04:41:01.780182 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.780093    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 04:41:01.784333 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784317    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 04:41:01.784333 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784332    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 04:41:01.784404 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784337    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 04:41:01.784404 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784340    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 04:41:01.784404 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784344    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 04:41:01.784404 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784347    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 04:41:01.784404 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784350    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 04:41:01.784404 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784352    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 04:41:01.784404 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784356    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 04:41:01.784404 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784359    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 04:41:01.784404 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784362    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 04:41:01.784404 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784365    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 04:41:01.784404 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784368    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 04:41:01.784404 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784372    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 04:41:01.784404 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784374    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 04:41:01.784404 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784377    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 04:41:01.784404 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784380    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 04:41:01.784404 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784383    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 04:41:01.784404 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784385    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 04:41:01.784404 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784388    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 04:41:01.784404 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784390    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 04:41:01.784404 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784393    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 04:41:01.784909 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784398    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 04:41:01.784909 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784401    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 04:41:01.784909 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784405    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 04:41:01.784909 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784410    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 04:41:01.784909 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784413    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 04:41:01.784909 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784417    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 04:41:01.784909 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784420    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 04:41:01.784909 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784423    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 04:41:01.784909 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784426    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 04:41:01.784909 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784429    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 04:41:01.784909 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784432    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 04:41:01.784909 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784435    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 04:41:01.784909 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784438    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 04:41:01.784909 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784440    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 04:41:01.784909 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784443    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 04:41:01.784909 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784445    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 04:41:01.784909 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784449    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 04:41:01.784909 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784452    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 04:41:01.784909 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784454    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 04:41:01.785374 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784457    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 04:41:01.785374 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784459    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 04:41:01.785374 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784461    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 04:41:01.785374 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784464    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 04:41:01.785374 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784467    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 04:41:01.785374 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784470    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 04:41:01.785374 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784473    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 04:41:01.785374 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784475    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 04:41:01.785374 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784478    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 04:41:01.785374 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784480    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 04:41:01.785374 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784483    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 04:41:01.785374 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784485    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 04:41:01.785374 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784488    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 04:41:01.785374 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784490    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 04:41:01.785374 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784494    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 04:41:01.785374 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784497    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 04:41:01.785374 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784500    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 04:41:01.785374 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784502    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 04:41:01.785374 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784505    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 04:41:01.785374 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784508    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 04:41:01.785890 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784510    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 04:41:01.785890 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784513    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 04:41:01.785890 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784515    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 04:41:01.785890 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784518    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 04:41:01.785890 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784520    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 04:41:01.785890 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784523    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 04:41:01.785890 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784526    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 04:41:01.785890 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784528    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 04:41:01.785890 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784531    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 04:41:01.785890 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784533    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 04:41:01.785890 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784537    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 04:41:01.785890 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784540    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 04:41:01.785890 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784542    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 04:41:01.785890 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784545    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 04:41:01.785890 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784547    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 04:41:01.785890 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784550    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 04:41:01.785890 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784553    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 04:41:01.785890 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784555    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 04:41:01.785890 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784558    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 04:41:01.786349 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784561    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 04:41:01.786349 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784563    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 04:41:01.786349 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784566    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 04:41:01.786349 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784570    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 04:41:01.786349 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784574    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 04:41:01.786349 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784576    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 04:41:01.786349 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784579    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 04:41:01.786349 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784583    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 04:41:01.786349 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784586    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 04:41:01.786349 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784589    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 04:41:01.786349 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784592    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 04:41:01.786349 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784594    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 04:41:01.786349 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784597    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 04:41:01.786349 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784600    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 04:41:01.786349 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784602    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 04:41:01.786349 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784605    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 04:41:01.786349 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784607    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 04:41:01.786349 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784610    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 04:41:01.786349 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.784612    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 04:41:01.786349 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785333    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 04:41:01.786837 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785340    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 04:41:01.786837 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785343    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 04:41:01.786837 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785347    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 04:41:01.786837 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785350    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 04:41:01.786837 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785353    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 04:41:01.786837 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785356    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 04:41:01.786837 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785358    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 04:41:01.786837 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785361    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 04:41:01.786837 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785364    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 04:41:01.786837 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785366    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 04:41:01.786837 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785369    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 04:41:01.786837 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785371    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 04:41:01.786837 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785375    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 04:41:01.786837 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785377    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 04:41:01.786837 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785380    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 04:41:01.786837 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785383    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 04:41:01.786837 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785387    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 04:41:01.786837 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785390    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 04:41:01.786837 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785393    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 04:41:01.786837 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785398    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 04:41:01.787391 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785401    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 04:41:01.787391 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785404    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 04:41:01.787391 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785406    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 04:41:01.787391 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785409    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 04:41:01.787391 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785412    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 04:41:01.787391 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785414    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 04:41:01.787391 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785417    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 04:41:01.787391 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785420    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 04:41:01.787391 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785422    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 04:41:01.787391 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785425    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 04:41:01.787391 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785428    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 04:41:01.787391 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785430    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 04:41:01.787391 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785433    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 04:41:01.787391 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785436    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 04:41:01.787391 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785439    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 04:41:01.787391 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785442    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 04:41:01.787391 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785444    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 04:41:01.787391 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785447    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 04:41:01.787391 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785450    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 04:41:01.787391 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785453    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 04:41:01.787908 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785456    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 04:41:01.787908 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785458    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 04:41:01.787908 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785461    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 04:41:01.787908 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785463    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 04:41:01.787908 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785466    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 04:41:01.787908 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785470    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 04:41:01.787908 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785473    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 04:41:01.787908 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785476    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 04:41:01.787908 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785479    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 04:41:01.787908 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785482    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 04:41:01.787908 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785484    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 04:41:01.787908 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785488    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 04:41:01.787908 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785490    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 04:41:01.787908 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785493    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 04:41:01.787908 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785495    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 04:41:01.787908 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785498    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 04:41:01.787908 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785501    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 04:41:01.787908 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785503    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 04:41:01.787908 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785506    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 04:41:01.788382 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785508    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 04:41:01.788382 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785511    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 04:41:01.788382 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785514    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 04:41:01.788382 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785516    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 04:41:01.788382 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785519    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 04:41:01.788382 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785521    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 04:41:01.788382 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785524    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 04:41:01.788382 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785526    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 04:41:01.788382 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785529    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 04:41:01.788382 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785531    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 04:41:01.788382 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785534    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 04:41:01.788382 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785538    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 04:41:01.788382 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785541    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 04:41:01.788382 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785543    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 04:41:01.788382 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785546    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 04:41:01.788382 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785548    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 04:41:01.788382 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785551    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 04:41:01.788382 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785554    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 04:41:01.788382 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785556    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 04:41:01.788382 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785559    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 04:41:01.788878 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785562    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 04:41:01.788878 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785566    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 04:41:01.788878 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785570    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 04:41:01.788878 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785572    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 04:41:01.788878 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785576    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 04:41:01.788878 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785579    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 04:41:01.788878 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785582    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 04:41:01.788878 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785585    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 04:41:01.788878 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785588    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 04:41:01.788878 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785591    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 04:41:01.788878 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785593    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 04:41:01.788878 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785596    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 04:41:01.788878 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785599    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 04:41:01.788878 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785601    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 04:41:01.788878 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785604    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 04:41:01.788878 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785606    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 04:41:01.788878 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785609    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 04:41:01.788878 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785612    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 04:41:01.788878 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.785614    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 04:41:01.789351 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786868    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 04:41:01.789351 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786876    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 04:41:01.789351 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786880    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 04:41:01.789351 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786883    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 04:41:01.789351 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786888    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 04:41:01.789351 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786892    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 04:41:01.789351 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786895    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 04:41:01.789351 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786898    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 04:41:01.789351 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786901    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 04:41:01.789351 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786904    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 04:41:01.789351 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786907    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 04:41:01.789351 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786911    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 04:41:01.789351 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786914    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 04:41:01.789351 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786917    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 04:41:01.789351 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786920    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 04:41:01.789351 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786923    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 04:41:01.789351 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786926    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 04:41:01.789351 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786929    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 04:41:01.789351 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786932    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 04:41:01.789867 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786935    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 04:41:01.789867 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786938    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 04:41:01.789867 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786941    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 04:41:01.789867 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786944    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 04:41:01.789867 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786947    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 04:41:01.789867 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786950    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 04:41:01.789867 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786952    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 04:41:01.789867 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786955    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 04:41:01.789867 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786957    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 04:41:01.789867 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786960    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 04:41:01.789867 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786962    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 04:41:01.789867 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786965    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 04:41:01.789867 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786967    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 04:41:01.789867 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786970    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 04:41:01.789867 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786972    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 04:41:01.789867 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786975    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 04:41:01.789867 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786977    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 04:41:01.789867 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786980    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 04:41:01.789867 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786983    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 04:41:01.789867 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786985    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 04:41:01.790362 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786988    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 04:41:01.790362 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786990    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 04:41:01.790362 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786993    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 04:41:01.790362 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786996    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 04:41:01.790362 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.786998    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 04:41:01.790362 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787001    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 04:41:01.790362 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787003    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 04:41:01.790362 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787006    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 04:41:01.790362 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787008    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 04:41:01.790362 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787011    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 04:41:01.790362 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787014    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 04:41:01.790362 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787017    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 04:41:01.790362 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787020    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 04:41:01.790362 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787022    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 04:41:01.790362 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787024    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 04:41:01.790362 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787027    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 04:41:01.790362 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787030    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 04:41:01.790362 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787032    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 04:41:01.790362 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787035    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 04:41:01.790362 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787037    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 04:41:01.790866 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787040    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 04:41:01.790866 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787042    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 04:41:01.790866 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787045    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 04:41:01.790866 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787048    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 04:41:01.790866 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787050    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 04:41:01.790866 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787053    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 04:41:01.790866 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787055    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 04:41:01.790866 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787058    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 04:41:01.790866 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787060    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 04:41:01.790866 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787062    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 04:41:01.790866 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787065    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 04:41:01.790866 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787069    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 04:41:01.790866 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787071    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 04:41:01.790866 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787074    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 04:41:01.790866 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787077    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 04:41:01.790866 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787079    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 04:41:01.790866 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787083    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 04:41:01.790866 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787086    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 04:41:01.790866 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787088    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 04:41:01.791352 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787091    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 04:41:01.791352 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787093    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 04:41:01.791352 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787096    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 04:41:01.791352 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787099    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 04:41:01.791352 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787101    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 04:41:01.791352 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787104    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 04:41:01.791352 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787106    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 04:41:01.791352 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787108    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 04:41:01.791352 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787111    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 04:41:01.791352 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787114    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 04:41:01.791352 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787116    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 04:41:01.791352 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787119    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 04:41:01.791352 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787121    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 04:41:01.791352 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787124    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 04:41:01.791352 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787127    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 04:41:01.791352 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787129    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 04:41:01.791352 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787132    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 04:41:01.791352 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787134    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 04:41:01.791352 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787137    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 04:41:01.791352 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787139    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787142    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787207    2575 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787241    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787253    2575 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787262    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787268    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787274    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787280    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787288    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787293    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787298    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787302    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787305    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787309    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787312    2575 flags.go:64] FLAG: --cgroup-root=""
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787315    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787318    2575 flags.go:64] FLAG: --client-ca-file=""
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787321    2575 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787324    2575 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787328    2575 flags.go:64] FLAG: --cluster-domain=""
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787331    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787334    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787337    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 04:41:01.791849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787340    2575 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787344    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787347    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787350    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787354    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787357    2575 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787359    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787362    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787366    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787369    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787373    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787376    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787378    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787381    2575 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787384    2575 flags.go:64] FLAG: --enable-server="true"
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787388    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787393    2575 flags.go:64] FLAG: --event-burst="100"
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787398    2575 flags.go:64] FLAG: --event-qps="50"
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787402    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787405    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787408    2575 flags.go:64] FLAG: --eviction-hard=""
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787411    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787414    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787419    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787422    2575 flags.go:64] FLAG: --eviction-soft=""
Sep 10 04:41:01.792470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787425    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787428    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787431    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787433    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787436    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787439    2575 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787442    2575 flags.go:64] FLAG: --feature-gates=""
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787446    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787449    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787452    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787456    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787459    2575 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787462    2575 flags.go:64] FLAG: --help="false"
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787465    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-132-70.ec2.internal"
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787468    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787471    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787474    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787477    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787480    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787483    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787486    2575 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787489    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787492    2575 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 04:41:01.793125 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787496    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787500    2575 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787503    2575 flags.go:64] FLAG: --kube-reserved=""
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787506    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787509    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787512    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787514    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787519    2575 flags.go:64] FLAG: --lock-file=""
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787522    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787525    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787528    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787533    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787536    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787539    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787541    2575 flags.go:64] FLAG: --logging-format="text"
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787544    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787547    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787550    2575 flags.go:64] FLAG: --manifest-url=""
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787553    2575 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787558    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787561    2575 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787565    2575 flags.go:64] FLAG: --max-pods="110"
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787568    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787571    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787575    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 04:41:01.793713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787578    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787581    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787584    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787587    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787595    2575 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787598    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787600    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787603    2575 flags.go:64] FLAG: --pod-cidr=""
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787609    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787614    2575 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787617    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787620    2575 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787623    2575 flags.go:64] FLAG: --port="10250"
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787626    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787630    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a86d01a69dc9e943"
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787633    2575 flags.go:64] FLAG: --qos-reserved=""
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787636    2575 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787639    2575 flags.go:64] FLAG: --register-node="true"
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787642    2575 flags.go:64] FLAG: --register-with-taints=""
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787646    2575 flags.go:64] FLAG: --registry-burst="10"
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787649    2575 flags.go:64] FLAG: --registry-qps="5"
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787651    2575 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787654    2575 flags.go:64] FLAG: --reserved-memory=""
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787658    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787661    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 04:41:01.794382 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787664    2575 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787667    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787670    2575 flags.go:64] FLAG: --runonce="false"
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787673    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787676    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787679    2575 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787681    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787685    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787688    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787691    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787694    2575 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787697    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787700    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787703    2575 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787706    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787708    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787713    2575 flags.go:64] FLAG: --system-cgroups=""
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787716    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787721    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787724    2575 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787727    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787732    2575 flags.go:64] FLAG: --tls-min-version=""
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787735    2575 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787737    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787740    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 04:41:01.795013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787743    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 04:41:01.795738 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787746    2575 flags.go:64] FLAG: --v="2"
Sep 10 04:41:01.795738 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787750    2575 flags.go:64] FLAG: --version="false"
Sep 10 04:41:01.795738 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787754    2575 flags.go:64] FLAG: --vmodule=""
Sep 10 04:41:01.795738 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787758    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 04:41:01.795738 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.787761    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 04:41:01.795738 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787907    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 04:41:01.795738 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787911    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 04:41:01.795738 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787914    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 04:41:01.795738 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787917    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 04:41:01.795738 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787920    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 04:41:01.795738 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787922    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 04:41:01.795738 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787925    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 04:41:01.795738 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787927    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 04:41:01.795738 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787932    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 04:41:01.795738 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787936    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 04:41:01.795738 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787939    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 04:41:01.795738 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787942    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 04:41:01.795738 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787945    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 04:41:01.795738 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787947    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 04:41:01.795738 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787950    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 04:41:01.796284 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787953    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 04:41:01.796284 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787956    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 04:41:01.796284 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787961    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 04:41:01.796284 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787971    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 04:41:01.796284 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787974    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 04:41:01.796284 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787976    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 04:41:01.796284 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787979    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 04:41:01.796284 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787983    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 04:41:01.796284 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787985    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 04:41:01.796284 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787988    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 04:41:01.796284 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787991    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 04:41:01.796284 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787993    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 04:41:01.796284 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787996    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 04:41:01.796284 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.787999    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 04:41:01.796284 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788001    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 04:41:01.796284 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788004    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 04:41:01.796284 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788006    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 04:41:01.796284 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788009    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 04:41:01.796284 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788011    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 04:41:01.796284 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788014    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 04:41:01.796769 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788017    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 04:41:01.796769 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788019    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 04:41:01.796769 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788022    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 04:41:01.796769 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788024    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 04:41:01.796769 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788027    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 04:41:01.796769 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788029    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 04:41:01.796769 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788032    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 04:41:01.796769 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788034    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 04:41:01.796769 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788037    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 04:41:01.796769 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788039    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 04:41:01.796769 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788042    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 04:41:01.796769 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788044    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 04:41:01.796769 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788047    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 04:41:01.796769 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788050    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 04:41:01.796769 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788054    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 04:41:01.796769 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788058    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 04:41:01.796769 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788061    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 04:41:01.796769 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788063    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 04:41:01.796769 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788066    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 04:41:01.797252 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788070    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 04:41:01.797252 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788072    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 04:41:01.797252 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788075    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 04:41:01.797252 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788078    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 04:41:01.797252 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788082    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 04:41:01.797252 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788085    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 04:41:01.797252 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788088    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 04:41:01.797252 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788091    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 04:41:01.797252 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788094    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 04:41:01.797252 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788096    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 04:41:01.797252 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788099    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 04:41:01.797252 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788102    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 04:41:01.797252 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788104    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 04:41:01.797252 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788107    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 04:41:01.797252 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788109    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 04:41:01.797252 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788112    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 04:41:01.797252 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788114    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 04:41:01.797252 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788117    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 04:41:01.797252 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788120    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 04:41:01.797252 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788122    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 04:41:01.797754 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788124    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 04:41:01.797754 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788127    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 04:41:01.797754 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788130    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 04:41:01.797754 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788133    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 04:41:01.797754 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788135    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 04:41:01.797754 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788137    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 04:41:01.797754 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788140    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 04:41:01.797754 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788144    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 04:41:01.797754 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788148    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 04:41:01.797754 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788151    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 04:41:01.797754 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788155    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 04:41:01.797754 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788158    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 04:41:01.797754 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788162    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 04:41:01.797754 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788165    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 04:41:01.797754 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788167    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 04:41:01.797754 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788170    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 04:41:01.797754 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788172    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 04:41:01.797754 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788175    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 04:41:01.797754 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788178    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 04:41:01.798320 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788181    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 04:41:01.798320 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788183    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 04:41:01.798320 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788186    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 04:41:01.798320 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788188    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 04:41:01.798320 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788191    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 04:41:01.798320 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.788193    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 04:41:01.798320 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.789097    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 04:41:01.798320 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.796922    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 04:41:01.798320 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.796939    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 04:41:01.798320 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.797898    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 04:41:01.798320 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.797906    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 04:41:01.798320 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.797975    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 04:41:01.798320 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.797981    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 04:41:01.798320 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.797985    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 04:41:01.798680 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.797988    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 04:41:01.798680 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.797991    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 04:41:01.798680 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.797994    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 04:41:01.798680 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.797997    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 04:41:01.798680 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798000    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 04:41:01.798680 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798002    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 04:41:01.798680 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798006    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 04:41:01.798680 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798010    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 04:41:01.798680 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798013    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 04:41:01.798680 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798022    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 04:41:01.798680 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798025    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 04:41:01.798680 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798028    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 04:41:01.798680 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798031    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 04:41:01.798680 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798034    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 04:41:01.798680 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798037    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 04:41:01.798680 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798040    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 04:41:01.798680 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798042    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 04:41:01.798680 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798048    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 04:41:01.798680 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798051    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 04:41:01.799159 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798053    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 04:41:01.799159 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798056    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 04:41:01.799159 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798058    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 04:41:01.799159 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798061    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 04:41:01.799159 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798064    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 04:41:01.799159 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798066    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 04:41:01.799159 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798069    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 04:41:01.799159 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798072    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 04:41:01.799159 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798075    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 04:41:01.799159 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798078    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 04:41:01.799159 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798081    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 04:41:01.799159 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798083    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 04:41:01.799159 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798086    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 04:41:01.799159 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798089    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 04:41:01.799159 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798091    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 04:41:01.799159 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798094    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 04:41:01.799159 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798096    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 04:41:01.799159 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798099    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 04:41:01.799159 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798101    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 04:41:01.799159 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798103    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 04:41:01.799159 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798106    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 04:41:01.799715 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798109    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 04:41:01.799715 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798111    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 04:41:01.799715 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798119    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 04:41:01.799715 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798122    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 04:41:01.799715 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798125    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 04:41:01.799715 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798127    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 04:41:01.799715 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798129    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 04:41:01.799715 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798132    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 04:41:01.799715 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798134    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 04:41:01.799715 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798137    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 04:41:01.799715 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798139    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 04:41:01.799715 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798142    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 04:41:01.799715 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798144    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 04:41:01.799715 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798147    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 04:41:01.799715 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798149    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 04:41:01.799715 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798152    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 04:41:01.799715 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798155    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 04:41:01.799715 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798157    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 04:41:01.799715 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798160    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 04:41:01.800192 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798163    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 04:41:01.800192 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798165    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 04:41:01.800192 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798168    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 04:41:01.800192 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798170    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 04:41:01.800192 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798173    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 04:41:01.800192 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798175    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 04:41:01.800192 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798178    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 04:41:01.800192 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798181    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 04:41:01.800192 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798183    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 04:41:01.800192 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798186    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 04:41:01.800192 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798188    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 04:41:01.800192 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798191    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 04:41:01.800192 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798193    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 04:41:01.800192 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798196    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 04:41:01.800192 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798198    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 04:41:01.800192 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798201    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 04:41:01.800192 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798209    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 04:41:01.800192 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798211    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 04:41:01.800192 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798214    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 04:41:01.800192 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798217    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 04:41:01.800192 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798219    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 04:41:01.800701 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798223    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 04:41:01.800701 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798226    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 04:41:01.800701 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798229    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 04:41:01.800701 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798231    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 04:41:01.800701 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798233    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 04:41:01.800701 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798236    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 04:41:01.800701 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798239    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 04:41:01.800701 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798241    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 04:41:01.800701 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798244    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 04:41:01.800701 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798247    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 04:41:01.800701 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798249    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 04:41:01.800701 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798252    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 04:41:01.800701 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798255    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 04:41:01.800701 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798259    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 04:41:01.800701 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798262    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 04:41:01.800701 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798265    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 04:41:01.801107 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.798270    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 04:41:01.801107 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798422    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 04:41:01.801107 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798428    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 04:41:01.801107 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798431    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 04:41:01.801107 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798434    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 04:41:01.801107 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798437    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 04:41:01.801107 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798440    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 04:41:01.801107 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798442    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 04:41:01.801107 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798445    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 04:41:01.801107 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798448    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 04:41:01.801107 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798451    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 04:41:01.801107 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798455    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 04:41:01.801107 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798463    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 04:41:01.801107 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798466    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 04:41:01.801449 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798469    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 04:41:01.801449 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798472    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 04:41:01.801449 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798476    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 04:41:01.801449 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798480    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 04:41:01.801449 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798483    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 04:41:01.801449 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798485    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 04:41:01.801449 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798488    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 04:41:01.801449 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798490    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 04:41:01.801449 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798493    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 04:41:01.801449 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798496    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 04:41:01.801449 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798499    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 04:41:01.801449 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798501    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 04:41:01.801449 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798504    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 04:41:01.801449 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798506    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 04:41:01.801449 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798509    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 04:41:01.801449 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798511    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 04:41:01.801449 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798514    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 04:41:01.801449 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798516    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 04:41:01.801449 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798519    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 04:41:01.801449 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798521    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 04:41:01.802043 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798523    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 04:41:01.802043 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798526    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 04:41:01.802043 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798528    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 04:41:01.802043 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798531    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 04:41:01.802043 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798533    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 04:41:01.802043 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798536    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 04:41:01.802043 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798538    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 04:41:01.802043 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798541    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 04:41:01.802043 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798543    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 04:41:01.802043 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798546    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 04:41:01.802043 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798548    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 04:41:01.802043 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798556    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 04:41:01.802043 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798559    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 04:41:01.802043 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798561    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 04:41:01.802043 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798564    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 04:41:01.802043 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798566    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 04:41:01.802043 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798569    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 04:41:01.802043 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798571    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 04:41:01.802043 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798573    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 04:41:01.802043 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798576    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 04:41:01.802534 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798579    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 04:41:01.802534 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798581    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 04:41:01.802534 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798584    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 04:41:01.802534 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798587    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 04:41:01.802534 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798590    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 04:41:01.802534 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798592    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 04:41:01.802534 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798595    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 04:41:01.802534 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798597    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 04:41:01.802534 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798600    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 04:41:01.802534 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798602    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 04:41:01.802534 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798605    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 04:41:01.802534 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798607    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 04:41:01.802534 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798609    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 04:41:01.802534 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798612    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 04:41:01.802534 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798614    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 04:41:01.802534 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798617    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 04:41:01.802534 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798619    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 04:41:01.802534 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798622    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 04:41:01.802534 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798624    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 04:41:01.802534 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798627    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 04:41:01.803037 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798629    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 04:41:01.803037 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798632    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 04:41:01.803037 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798634    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 04:41:01.803037 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798637    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 04:41:01.803037 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798645    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 04:41:01.803037 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798647    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 04:41:01.803037 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798650    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 04:41:01.803037 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798652    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 04:41:01.803037 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798656    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 04:41:01.803037 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798660    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 04:41:01.803037 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798663    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 04:41:01.803037 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798666    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 04:41:01.803037 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798668    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 04:41:01.803037 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798678    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 04:41:01.803037 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798682    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 04:41:01.803037 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798685    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 04:41:01.803037 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798688    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 04:41:01.803037 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798690    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 04:41:01.803037 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798693    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 04:41:01.803483 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798695    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 04:41:01.803483 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798698    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 04:41:01.803483 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798701    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 04:41:01.803483 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798703    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 04:41:01.803483 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798705    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 04:41:01.803483 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798708    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 04:41:01.803483 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:01.798710    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 04:41:01.803483 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.798715    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 04:41:01.803483 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.798848    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 04:41:01.803483 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.801409    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 04:41:01.803483 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.802450    2575 server.go:1019] "Starting client certificate rotation"
Sep 10 04:41:01.803483 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.802567    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 04:41:01.803483 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.802611    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 04:41:01.806060 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.806045    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 04:41:01.807741 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.807711    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 04:41:01.817196 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.817176    2575 log.go:25] "Validated CRI v1 runtime API"
Sep 10 04:41:01.820399 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.820380    2575 log.go:25] "Validated CRI v1 image API"
Sep 10 04:41:01.821237 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.821220    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 04:41:01.825732 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.825709    2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9ab21d74-a765-4296-91ea-fe2ac6c1b145:/dev/nvme0n1p4 d98e1046-de88-4520-a120-3f20db55bec2:/dev/nvme0n1p3]
Sep 10 04:41:01.825806 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.825731    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 04:41:01.831957 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.831848    2575 manager.go:216] Machine: {Timestamp:2026-09-10 04:41:01.829440064 +0000 UTC m=+0.366460435 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3040860 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2af47508bfdb54a78122df5fa4f63e SystemUUID:ec2af475-08bf-db54-a781-22df5fa4f63e BootID:37c235fd-6003-4182-b797-9076235f0bfd Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8e:33:23:0b:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8e:33:23:0b:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:f0:ff:70:f0:74 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 04:41:01.831957 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.831955    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 04:41:01.832061 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.832042    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 04:41:01.834136 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.834120    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 04:41:01.835632 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.835609    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 04:41:01.835841 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.835636    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-70.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 04:41:01.836538 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.836526    2575 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 04:41:01.836576 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.836541    2575 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 04:41:01.836576 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.836550    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 04:41:01.837365 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.837353    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 04:41:01.839091 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.839080    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 04:41:01.843343 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.843327    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 04:41:01.843468 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.843457    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 04:41:01.843505 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.843491    2575 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 04:41:01.843540 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.843505    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 04:41:01.843540 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.843529    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 04:41:01.843600 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.843542    2575 kubelet.go:398] "Adding apiserver pod source"
Sep 10 04:41:01.843600 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.843554    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 04:41:01.844000 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.843983    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 04:41:01.844055 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.844008    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 04:41:01.846448 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.846427    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 04:41:01.849329 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.849312    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 04:41:01.849399 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.849345    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 04:41:01.849501 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.849492    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 04:41:01.849537 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.849507    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 04:41:01.849537 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.849513    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 04:41:01.849537 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.849518    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 04:41:01.849537 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.849526    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 04:41:01.849537 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.849531    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 04:41:01.850160 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.850150    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 04:41:01.850203 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.850160    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 04:41:01.850203 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.850171    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 04:41:01.850203 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.850179    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 04:41:01.850203 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.850191    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 04:41:01.850203 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.850202    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 04:41:01.851095 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.851081    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 04:41:01.851095 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.851094    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 04:41:01.854949 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.854900    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-70.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 04:41:01.855031 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:01.854973    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-70.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 04:41:01.855096 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:01.855053    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 04:41:01.855161 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.855149    2575 server.go:1268] "Started kubelet"
Sep 10 04:41:01.855216 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.855193    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 04:41:01.855293 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.855246    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 04:41:01.855364 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.855319    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 04:41:01.859525 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.859502    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 04:41:01.859846 ip-10-0-132-70 systemd[1]: Started Kubernetes Kubelet.
Sep 10 04:41:01.862137 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.860571    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nwkwv"
Sep 10 04:41:01.862137 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.860866    2575 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 04:41:01.866484 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:01.866460    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 04:41:01.866484 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.866472    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nwkwv"
Sep 10 04:41:01.869011 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:01.867908    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-70.ec2.internal.18d3dbe6eca08a1c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-70.ec2.internal,UID:ip-10-0-132-70.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-70.ec2.internal,},FirstTimestamp:2026-09-10 04:41:01.855123996 +0000 UTC m=+0.392144369,LastTimestamp:2026-09-10 04:41:01.855123996 +0000 UTC m=+0.392144369,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-70.ec2.internal,}"
Sep 10 04:41:01.978875 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.978838    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 04:41:01.979477 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.979462    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 04:41:01.979690 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:01.979666    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-70.ec2.internal\" not found"
Sep 10 04:41:01.980302 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.980284    2575 factory.go:55] Registering systemd factory
Sep 10 04:41:01.980302 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.980302    2575 factory.go:223] Registration of the systemd container factory successfully
Sep 10 04:41:01.980446 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.980306    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 04:41:01.980446 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.980309    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 04:41:01.980446 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.980332    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 04:41:01.980446 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.980422    2575 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 04:41:01.980446 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.980429    2575 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 04:41:01.980639 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.980536    2575 factory.go:153] Registering CRI-O factory
Sep 10 04:41:01.980639 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.980550    2575 factory.go:223] Registration of the crio container factory successfully
Sep 10 04:41:01.980639 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.980600    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 04:41:01.980639 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.980615    2575 factory.go:103] Registering Raw factory
Sep 10 04:41:01.980639 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.980625    2575 manager.go:1196] Started watching for new ooms in manager
Sep 10 04:41:01.981063 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.981048    2575 manager.go:349] Starting recovery of all containers
Sep 10 04:41:01.981722 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:01.981682    2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-70.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 04:41:01.981893 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:01.981869    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 04:41:01.991253 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.991229    2575 manager.go:354] Recovery completed
Sep 10 04:41:01.995259 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.995247    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 04:41:01.997261 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.997244    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-70.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 04:41:01.997342 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.997273    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-70.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 04:41:01.997342 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.997283    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-70.ec2.internal" event="NodeHasSufficientPID"
Sep 10 04:41:01.997887 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.997872    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 04:41:01.997887 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.997886    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 04:41:01.997992 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.997878    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 04:41:01.997992 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.997906    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 04:41:01.998597 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.998553    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-70.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 04:41:01.998597 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.998582    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-70.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 04:41:01.998597 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.998595    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-70.ec2.internal" event="NodeHasSufficientPID"
Sep 10 04:41:01.999241 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.999227    2575 policy_none.go:49] "None policy: Start"
Sep 10 04:41:01.999309 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.999244    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 04:41:01.999309 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:01.999258    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 04:41:02.000412 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.000396    2575 policy_none.go:47] "Start"
Sep 10 04:41:02.035386 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.035371    2575 manager.go:337] "Starting Device Plugin manager"
Sep 10 04:41:02.044745 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:02.035416    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 04:41:02.044745 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.035430    2575 server.go:87] "Starting device plugin registration server"
Sep 10 04:41:02.044745 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.035754    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 04:41:02.044745 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.035765    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 04:41:02.044745 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.035934    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 04:41:02.044745 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.036007    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 04:41:02.044745 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.036016    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 04:41:02.044745 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:02.036591    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 04:41:02.044745 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:02.036634    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-70.ec2.internal\" not found"
Sep 10 04:41:02.079720 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.079681    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 04:41:02.080943 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.080923    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 04:41:02.081003 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.080993    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 04:41:02.081054 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.081027    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 04:41:02.081154 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:02.081126    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 04:41:02.083494 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.083474    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 04:41:02.136425 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.136390    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 04:41:02.137833 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.137798    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-70.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 04:41:02.137916 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.137853    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-70.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 04:41:02.137916 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.137865    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-70.ec2.internal" event="NodeHasSufficientPID"
Sep 10 04:41:02.137916 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.137894    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-70.ec2.internal"
Sep 10 04:41:02.146010 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.145978    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-70.ec2.internal"
Sep 10 04:41:02.146010 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:02.146008    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-70.ec2.internal\": node \"ip-10-0-132-70.ec2.internal\" not found"
Sep 10 04:41:02.159964 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:02.159944    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-70.ec2.internal\" not found"
Sep 10 04:41:02.181204 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.181176    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-70.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-70.ec2.internal"]
Sep 10 04:41:02.181320 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.181244    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 04:41:02.182272 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.182255    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-70.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 04:41:02.182370 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.182289    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-70.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 04:41:02.182370 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.182303    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-70.ec2.internal" event="NodeHasSufficientPID"
Sep 10 04:41:02.183560 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.183544    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 04:41:02.183736 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.183721    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-70.ec2.internal"
Sep 10 04:41:02.183783 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.183749    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 04:41:02.184328 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.184312    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-70.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 04:41:02.184328 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.184322    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-70.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 04:41:02.184418 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.184340    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-70.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 04:41:02.184418 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.184342    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-70.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 04:41:02.184418 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.184360    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-70.ec2.internal" event="NodeHasSufficientPID"
Sep 10 04:41:02.184418 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.184351    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-70.ec2.internal" event="NodeHasSufficientPID"
Sep 10 04:41:02.185391 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.185376    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-70.ec2.internal"
Sep 10 04:41:02.185457 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.185401    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 04:41:02.186074 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.186059    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-70.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 04:41:02.186127 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.186085    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-70.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 04:41:02.186127 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.186098    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-70.ec2.internal" event="NodeHasSufficientPID"
Sep 10 04:41:02.213579 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:02.213556    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-70.ec2.internal\" not found" node="ip-10-0-132-70.ec2.internal"
Sep 10 04:41:02.218077 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:02.218062    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-70.ec2.internal\" not found" node="ip-10-0-132-70.ec2.internal"
Sep 10 04:41:02.260026 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:02.259998    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-70.ec2.internal\" not found"
Sep 10 04:41:02.281707 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.281682    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27bd344442d49c85b9c1f8743a1a1f9f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-70.ec2.internal\" (UID: \"27bd344442d49c85b9c1f8743a1a1f9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-70.ec2.internal"
Sep 10 04:41:02.281794 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.281712    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27bd344442d49c85b9c1f8743a1a1f9f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-70.ec2.internal\" (UID: \"27bd344442d49c85b9c1f8743a1a1f9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-70.ec2.internal"
Sep 10 04:41:02.281794 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.281732    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9d5f561d6e275e920a4621f246ba503e-config\") pod \"kube-apiserver-proxy-ip-10-0-132-70.ec2.internal\" (UID: \"9d5f561d6e275e920a4621f246ba503e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-70.ec2.internal"
Sep 10 04:41:02.360424 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:02.360352    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-70.ec2.internal\" not found"
Sep 10 04:41:02.382864 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.382838    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9d5f561d6e275e920a4621f246ba503e-config\") pod \"kube-apiserver-proxy-ip-10-0-132-70.ec2.internal\" (UID: \"9d5f561d6e275e920a4621f246ba503e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-70.ec2.internal"
Sep 10 04:41:02.383014 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.382867    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9d5f561d6e275e920a4621f246ba503e-config\") pod \"kube-apiserver-proxy-ip-10-0-132-70.ec2.internal\" (UID: \"9d5f561d6e275e920a4621f246ba503e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-70.ec2.internal"
Sep 10 04:41:02.383014 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.382903    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27bd344442d49c85b9c1f8743a1a1f9f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-70.ec2.internal\" (UID: \"27bd344442d49c85b9c1f8743a1a1f9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-70.ec2.internal"
Sep 10 04:41:02.383014 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.382873    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27bd344442d49c85b9c1f8743a1a1f9f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-70.ec2.internal\" (UID: \"27bd344442d49c85b9c1f8743a1a1f9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-70.ec2.internal"
Sep 10 04:41:02.383014 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.382930    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27bd344442d49c85b9c1f8743a1a1f9f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-70.ec2.internal\" (UID: \"27bd344442d49c85b9c1f8743a1a1f9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-70.ec2.internal"
Sep 10 04:41:02.383014 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:02.382970    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27bd344442d49c85b9c1f8743a1a1f9f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-70.ec2.internal\" (UID: \"27bd344442d49c85b9c1f8743a1a1f9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-70.ec2.internal"
Sep 10 04:41:04.236276 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:04.236225    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-70.ec2.internal\" not found"
Sep 10 04:41:04.291803 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:04.291768    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-70.ec2.internal"
Sep 10 04:41:04.296365 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:04.296346    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-70.ec2.internal"
Sep 10 04:41:04.336940 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:04.336910    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-70.ec2.internal\" not found"
Sep 10 04:41:04.437583 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:04.437496    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-70.ec2.internal\" not found"
Sep 10 04:41:04.504526 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:04.504500    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 04:41:04.538381 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:04.538352    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-70.ec2.internal\" not found"
Sep 10 04:41:04.577924 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:04.577892    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 04:41:04.578325 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:04.578035    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 04:41:04.578325 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:04.578095    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 04:41:04.638658 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:04.638630    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-70.ec2.internal\" not found"
Sep 10 04:41:04.647200 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:04.647165    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 04:36:03 +0000 UTC" deadline="2028-05-27 21:49:07.016112234 +0000 UTC"
Sep 10 04:41:04.647200 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:04.647198    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15017h8m2.368917481s"
Sep 10 04:41:04.716336 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:04.716268    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 04:41:04.731295 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:04.731269    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 04:41:04.755188 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:04.755155    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-70.ec2.internal"
Sep 10 04:41:04.755391 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:04.755170    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 04:41:04.766176 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:04.766154    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 04:41:04.767188 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:04.767174    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-70.ec2.internal"
Sep 10 04:41:04.768453 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:04.768436    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 04:41:04.773391 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:04.773366    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 04:41:04.789324 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:04.789301    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5m87z"
Sep 10 04:41:04.798594 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:04.798569    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5m87z"
Sep 10 04:41:04.829740 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:04.829709    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5f561d6e275e920a4621f246ba503e.slice/crio-0d6627497a63ecd9cf77b18649c5ba85e212caa028a01b50adf585293e732531 WatchSource:0}: Error finding container 0d6627497a63ecd9cf77b18649c5ba85e212caa028a01b50adf585293e732531: Status 404 returned error can't find the container with id 0d6627497a63ecd9cf77b18649c5ba85e212caa028a01b50adf585293e732531
Sep 10 04:41:04.830002 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:04.829984    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27bd344442d49c85b9c1f8743a1a1f9f.slice/crio-b38e76ad34cad8a677094c56593734c2901243e666232e63f0769f27a0671c0c WatchSource:0}: Error finding container b38e76ad34cad8a677094c56593734c2901243e666232e63f0769f27a0671c0c: Status 404 returned error can't find the container with id b38e76ad34cad8a677094c56593734c2901243e666232e63f0769f27a0671c0c
Sep 10 04:41:04.835387 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:04.835372    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 04:41:04.859488 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:04.859442    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-70.ec2.internal" event={"ID":"27bd344442d49c85b9c1f8743a1a1f9f","Type":"ContainerStarted","Data":"b38e76ad34cad8a677094c56593734c2901243e666232e63f0769f27a0671c0c"}
Sep 10 04:41:04.860479 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:04.860453    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-70.ec2.internal" event={"ID":"9d5f561d6e275e920a4621f246ba503e","Type":"ContainerStarted","Data":"0d6627497a63ecd9cf77b18649c5ba85e212caa028a01b50adf585293e732531"}
Sep 10 04:41:05.625878 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.625845    2575 apiserver.go:52] "Watching apiserver"
Sep 10 04:41:05.634111 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.634083    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 04:41:05.635524 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.635495    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-8r87c","kube-system/kube-apiserver-proxy-ip-10-0-132-70.ec2.internal","openshift-multus/multus-additional-cni-plugins-ms5c5","openshift-multus/network-metrics-daemon-7krfk","openshift-network-diagnostics/network-check-target-47tk5","openshift-network-operator/iptables-alerter-wg9t7","openshift-ovn-kubernetes/ovnkube-node-j2hd2","kube-system/global-pull-secret-syncer-pclmn","kube-system/konnectivity-agent-c69jd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr","openshift-cluster-node-tuning-operator/tuned-psjfv","openshift-dns/node-resolver-2kswj","openshift-image-registry/node-ca-n2zlm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-70.ec2.internal"]
Sep 10 04:41:05.637310 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.637285    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c69jd"
Sep 10 04:41:05.638641 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.638618    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2kswj"
Sep 10 04:41:05.640277 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.640242    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vhk75\""
Sep 10 04:41:05.640277 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.640265    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 04:41:05.640426 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.640242    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 04:41:05.641202 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.640759    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 04:41:05.641202 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.641109    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m66hn\""
Sep 10 04:41:05.641445 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.641423    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.641774 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.641756    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 04:41:05.642616 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.642599    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.643622 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.643598    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 04:41:05.643952 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.643928    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:05.644045 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.643979    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 04:41:05.644045 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:05.644031    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47tk5" podUID="7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8"
Sep 10 04:41:05.644297 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.644275    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s677c\""
Sep 10 04:41:05.644389 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.644305    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 04:41:05.644520 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.644505    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 04:41:05.644579 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.644534    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 04:41:05.645086 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.644905    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mdnb2\""
Sep 10 04:41:05.645176 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.645123    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 04:41:05.645701 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.645388    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wg9t7"
Sep 10 04:41:05.646781 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.646761    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.648078 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.647964    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hwcjp\""
Sep 10 04:41:05.648281 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.648250    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 04:41:05.648521 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.648503    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:05.648600 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:05.648578    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7krfk" podUID="4bbb6665-dfae-48eb-9f23-f5147f2ccebb"
Sep 10 04:41:05.649663 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.649506    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 04:41:05.649663 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.649580    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 04:41:05.649877 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.649856    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 04:41:05.649963 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.649913    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 04:41:05.650018 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.649857    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 04:41:05.650306 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.650147    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j4jxp\""
Sep 10 04:41:05.650306 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.650175    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:05.650306 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:05.650275    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclmn" podUID="920551bb-5e01-45e3-80cd-87f46761fb29"
Sep 10 04:41:05.650684 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.650669    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 04:41:05.650745 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.650695    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 04:41:05.650945 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.650885    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 04:41:05.651771 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.651732    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:05.653213 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.653192    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.654625 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.654605    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n2zlm"
Sep 10 04:41:05.655233 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.655215    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 04:41:05.655649 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.655633    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 04:41:05.655724 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.655686    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 04:41:05.655845 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.655818    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cqf2m\""
Sep 10 04:41:05.655939 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.655636    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 04:41:05.658396 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.656934    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7k27c\""
Sep 10 04:41:05.658396 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.657473    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 04:41:05.658396 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.658056    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 04:41:05.658396 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.658163    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 04:41:05.658773 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.658596    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7c7zc\""
Sep 10 04:41:05.658773 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.658673    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 04:41:05.661540 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.661523    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 04:41:05.666079 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.665985    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dafe7236-9f57-4f88-97d5-b30b2f8ee269-cni-binary-copy\") pod \"multus-additional-cni-plugins-ms5c5\" (UID: \"dafe7236-9f57-4f88-97d5-b30b2f8ee269\") " pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.666079 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666016    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dafe7236-9f57-4f88-97d5-b30b2f8ee269-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ms5c5\" (UID: \"dafe7236-9f57-4f88-97d5-b30b2f8ee269\") " pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.666079 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666044    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-run-systemd\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.666274 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666082    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-etc-sysctl-d\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.666274 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666107    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/47240839-28f9-4291-b91f-b22c142ce1f0-iptables-alerter-script\") pod \"iptables-alerter-wg9t7\" (UID: \"47240839-28f9-4291-b91f-b22c142ce1f0\") " pod="openshift-network-operator/iptables-alerter-wg9t7"
Sep 10 04:41:05.666274 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666131    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m69s\" (UniqueName: \"kubernetes.io/projected/dafe7236-9f57-4f88-97d5-b30b2f8ee269-kube-api-access-9m69s\") pod \"multus-additional-cni-plugins-ms5c5\" (UID: \"dafe7236-9f57-4f88-97d5-b30b2f8ee269\") " pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.666274 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666153    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02eb37ae-67ce-4454-b34a-e3666a73f999-serviceca\") pod \"node-ca-n2zlm\" (UID: \"02eb37ae-67ce-4454-b34a-e3666a73f999\") " pod="openshift-image-registry/node-ca-n2zlm"
Sep 10 04:41:05.666274 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666175    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-host-var-lib-cni-bin\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.666274 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666200    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a75e8f96-1b83-4636-9a7d-57a5892e5e2b-socket-dir\") pod \"aws-ebs-csi-driver-node-b78sr\" (UID: \"a75e8f96-1b83-4636-9a7d-57a5892e5e2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:05.666274 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666237    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a75e8f96-1b83-4636-9a7d-57a5892e5e2b-sys-fs\") pod \"aws-ebs-csi-driver-node-b78sr\" (UID: \"a75e8f96-1b83-4636-9a7d-57a5892e5e2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:05.666274 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666261    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-etc-openvswitch\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.666657 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666285    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-log-socket\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.666657 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666306    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-host-cni-netd\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.666657 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666330    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9d909143-4c70-4fe1-900a-3cd520db32e5-ovnkube-script-lib\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.666657 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666353    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8fa349bb-d0f7-48c7-ba38-d533479eb889-multus-daemon-config\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.666657 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666375    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a1db6ec-8b9d-4f0a-8bf0-de72e290cdd1-tmp-dir\") pod \"node-resolver-2kswj\" (UID: \"1a1db6ec-8b9d-4f0a-8bf0-de72e290cdd1\") " pod="openshift-dns/node-resolver-2kswj"
Sep 10 04:41:05.666657 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666398    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9d909143-4c70-4fe1-900a-3cd520db32e5-env-overrides\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.666657 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666420    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-etc-sysconfig\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.666657 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666439    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-lib-modules\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.666657 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666452    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4342bc6-a78e-4341-bdd6-c9e991c1c131-etc-tuned\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.666657 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666466    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-multus-cni-dir\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.666657 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666486    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-host-var-lib-cni-multus\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.666657 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666503    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-hostroot\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.666657 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666581    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-etc-kubernetes\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.666657 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666620    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-etc-sysctl-conf\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.666657 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666649    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-run\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.667371 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666674    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7fc6a301-6f28-4a4f-b547-2991c1cbb8b6-konnectivity-ca\") pod \"konnectivity-agent-c69jd\" (UID: \"7fc6a301-6f28-4a4f-b547-2991c1cbb8b6\") " pod="kube-system/konnectivity-agent-c69jd"
Sep 10 04:41:05.667371 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666707    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d909143-4c70-4fe1-900a-3cd520db32e5-ovn-node-metrics-cert\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.667371 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666738    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-run-openvswitch\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.667371 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666773    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-host-cni-bin\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.667371 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666804    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-host-run-multus-certs\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.667371 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666834    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a75e8f96-1b83-4636-9a7d-57a5892e5e2b-etc-selinux\") pod \"aws-ebs-csi-driver-node-b78sr\" (UID: \"a75e8f96-1b83-4636-9a7d-57a5892e5e2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:05.667371 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666855    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a1db6ec-8b9d-4f0a-8bf0-de72e290cdd1-hosts-file\") pod \"node-resolver-2kswj\" (UID: \"1a1db6ec-8b9d-4f0a-8bf0-de72e290cdd1\") " pod="openshift-dns/node-resolver-2kswj"
Sep 10 04:41:05.667371 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666878    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-host-var-lib-kubelet\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.667371 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666918    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a75e8f96-1b83-4636-9a7d-57a5892e5e2b-device-dir\") pod \"aws-ebs-csi-driver-node-b78sr\" (UID: \"a75e8f96-1b83-4636-9a7d-57a5892e5e2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:05.667371 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666941    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dafe7236-9f57-4f88-97d5-b30b2f8ee269-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ms5c5\" (UID: \"dafe7236-9f57-4f88-97d5-b30b2f8ee269\") " pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.667371 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666969    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-var-lib-openvswitch\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.667371 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.666991    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret\") pod \"global-pull-secret-syncer-pclmn\" (UID: \"920551bb-5e01-45e3-80cd-87f46761fb29\") " pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:05.667371 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667025    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-etc-modprobe-d\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.667371 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667049    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-host\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.667371 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667086    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6qcw\" (UniqueName: \"kubernetes.io/projected/c4342bc6-a78e-4341-bdd6-c9e991c1c131-kube-api-access-b6qcw\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.667371 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667122    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8fa349bb-d0f7-48c7-ba38-d533479eb889-cni-binary-copy\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.668123 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667150    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/920551bb-5e01-45e3-80cd-87f46761fb29-kubelet-config\") pod \"global-pull-secret-syncer-pclmn\" (UID: \"920551bb-5e01-45e3-80cd-87f46761fb29\") " pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:05.668123 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667172    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-system-cni-dir\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.668123 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667195    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-host-run-k8s-cni-cncf-io\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.668123 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667230    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-host-run-netns\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.668123 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667261    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7hpx\" (UniqueName: \"kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx\") pod \"network-check-target-47tk5\" (UID: \"7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8\") " pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:05.668123 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667287    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47240839-28f9-4291-b91f-b22c142ce1f0-host-slash\") pod \"iptables-alerter-wg9t7\" (UID: \"47240839-28f9-4291-b91f-b22c142ce1f0\") " pod="openshift-network-operator/iptables-alerter-wg9t7"
Sep 10 04:41:05.668123 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667308    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a75e8f96-1b83-4636-9a7d-57a5892e5e2b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b78sr\" (UID: \"a75e8f96-1b83-4636-9a7d-57a5892e5e2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:05.668123 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667331    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-systemd-units\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.668123 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667352    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs\") pod \"network-metrics-daemon-7krfk\" (UID: \"4bbb6665-dfae-48eb-9f23-f5147f2ccebb\") " pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:05.668123 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667374    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-var-lib-kubelet\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.668123 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667396    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85bjj\" (UniqueName: \"kubernetes.io/projected/47240839-28f9-4291-b91f-b22c142ce1f0-kube-api-access-85bjj\") pod \"iptables-alerter-wg9t7\" (UID: \"47240839-28f9-4291-b91f-b22c142ce1f0\") " pod="openshift-network-operator/iptables-alerter-wg9t7"
Sep 10 04:41:05.668123 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667415    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9d909143-4c70-4fe1-900a-3cd520db32e5-ovnkube-config\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.668123 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667429    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a75e8f96-1b83-4636-9a7d-57a5892e5e2b-registration-dir\") pod \"aws-ebs-csi-driver-node-b78sr\" (UID: \"a75e8f96-1b83-4636-9a7d-57a5892e5e2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:05.668123 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667462    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-host-run-netns\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.668123 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667493    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-node-log\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.668123 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667522    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-host-run-ovn-kubernetes\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.668922 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667545    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-etc-systemd\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.668922 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667572    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02eb37ae-67ce-4454-b34a-e3666a73f999-host\") pod \"node-ca-n2zlm\" (UID: \"02eb37ae-67ce-4454-b34a-e3666a73f999\") " pod="openshift-image-registry/node-ca-n2zlm"
Sep 10 04:41:05.668922 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667604    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5n6v\" (UniqueName: \"kubernetes.io/projected/02eb37ae-67ce-4454-b34a-e3666a73f999-kube-api-access-p5n6v\") pod \"node-ca-n2zlm\" (UID: \"02eb37ae-67ce-4454-b34a-e3666a73f999\") " pod="openshift-image-registry/node-ca-n2zlm"
Sep 10 04:41:05.668922 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667634    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k74vc\" (UniqueName: \"kubernetes.io/projected/1a1db6ec-8b9d-4f0a-8bf0-de72e290cdd1-kube-api-access-k74vc\") pod \"node-resolver-2kswj\" (UID: \"1a1db6ec-8b9d-4f0a-8bf0-de72e290cdd1\") " pod="openshift-dns/node-resolver-2kswj"
Sep 10 04:41:05.668922 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667667    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-host-kubelet\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.668922 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667695    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4342bc6-a78e-4341-bdd6-c9e991c1c131-tmp\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.668922 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667721    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpph5\" (UniqueName: \"kubernetes.io/projected/a75e8f96-1b83-4636-9a7d-57a5892e5e2b-kube-api-access-lpph5\") pod \"aws-ebs-csi-driver-node-b78sr\" (UID: \"a75e8f96-1b83-4636-9a7d-57a5892e5e2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:05.668922 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667746    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dafe7236-9f57-4f88-97d5-b30b2f8ee269-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ms5c5\" (UID: \"dafe7236-9f57-4f88-97d5-b30b2f8ee269\") " pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.668922 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667768    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-host-slash\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.668922 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667791    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-run-ovn\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.668922 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667809    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.668922 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667831    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g96d4\" (UniqueName: \"kubernetes.io/projected/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-kube-api-access-g96d4\") pod \"network-metrics-daemon-7krfk\" (UID: \"4bbb6665-dfae-48eb-9f23-f5147f2ccebb\") " pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:05.668922 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667861    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-multus-conf-dir\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.668922 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667884    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-etc-kubernetes\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.668922 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667904    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dafe7236-9f57-4f88-97d5-b30b2f8ee269-system-cni-dir\") pod \"multus-additional-cni-plugins-ms5c5\" (UID: \"dafe7236-9f57-4f88-97d5-b30b2f8ee269\") " pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.668922 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667918    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dafe7236-9f57-4f88-97d5-b30b2f8ee269-cnibin\") pod \"multus-additional-cni-plugins-ms5c5\" (UID: \"dafe7236-9f57-4f88-97d5-b30b2f8ee269\") " pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.669738 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667947    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrg95\" (UniqueName: \"kubernetes.io/projected/9d909143-4c70-4fe1-900a-3cd520db32e5-kube-api-access-jrg95\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.669738 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.667982    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/920551bb-5e01-45e3-80cd-87f46761fb29-dbus\") pod \"global-pull-secret-syncer-pclmn\" (UID: \"920551bb-5e01-45e3-80cd-87f46761fb29\") " pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:05.669738 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.668020    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-sys\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.669738 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.668047    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-cnibin\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.669738 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.668085    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb4s5\" (UniqueName: \"kubernetes.io/projected/8fa349bb-d0f7-48c7-ba38-d533479eb889-kube-api-access-rb4s5\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.669738 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.668113    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7fc6a301-6f28-4a4f-b547-2991c1cbb8b6-agent-certs\") pod \"konnectivity-agent-c69jd\" (UID: \"7fc6a301-6f28-4a4f-b547-2991c1cbb8b6\") " pod="kube-system/konnectivity-agent-c69jd"
Sep 10 04:41:05.669738 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.668139    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-multus-socket-dir-parent\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.768694 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.768653    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a1db6ec-8b9d-4f0a-8bf0-de72e290cdd1-tmp-dir\") pod \"node-resolver-2kswj\" (UID: \"1a1db6ec-8b9d-4f0a-8bf0-de72e290cdd1\") " pod="openshift-dns/node-resolver-2kswj"
Sep 10 04:41:05.768694 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.768683    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9d909143-4c70-4fe1-900a-3cd520db32e5-env-overrides\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.768694 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.768699    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-etc-sysconfig\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.768961 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.768720    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-lib-modules\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.768961 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.768742    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4342bc6-a78e-4341-bdd6-c9e991c1c131-etc-tuned\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.768961 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.768807    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-etc-sysconfig\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.768961 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.768854    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-multus-cni-dir\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.768961 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.768882    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-host-var-lib-cni-multus\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.768961 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.768907    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-hostroot\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.768961 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.768931    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-etc-kubernetes\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.768961 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.768943    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-multus-cni-dir\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.768961 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.768951    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-lib-modules\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.768961 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.768957    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-etc-sysctl-conf\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.769721 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.768999    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-run\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.769721 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769011    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a1db6ec-8b9d-4f0a-8bf0-de72e290cdd1-tmp-dir\") pod \"node-resolver-2kswj\" (UID: \"1a1db6ec-8b9d-4f0a-8bf0-de72e290cdd1\") " pod="openshift-dns/node-resolver-2kswj"
Sep 10 04:41:05.769721 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769017    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-host-var-lib-cni-multus\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.769721 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769041    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7fc6a301-6f28-4a4f-b547-2991c1cbb8b6-konnectivity-ca\") pod \"konnectivity-agent-c69jd\" (UID: \"7fc6a301-6f28-4a4f-b547-2991c1cbb8b6\") " pod="kube-system/konnectivity-agent-c69jd"
Sep 10 04:41:05.769721 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769086    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-hostroot\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.769721 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769092    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d909143-4c70-4fe1-900a-3cd520db32e5-ovn-node-metrics-cert\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.769721 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769129    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-etc-sysctl-conf\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.769721 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769142    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-run\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.769721 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769141    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-run-openvswitch\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.769721 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769186    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-run-openvswitch\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.769721 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769201    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-host-cni-bin\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.769721 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769205    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-etc-kubernetes\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.769721 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769228    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-host-run-multus-certs\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.769721 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769247    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9d909143-4c70-4fe1-900a-3cd520db32e5-env-overrides\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.769721 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769256    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a75e8f96-1b83-4636-9a7d-57a5892e5e2b-etc-selinux\") pod \"aws-ebs-csi-driver-node-b78sr\" (UID: \"a75e8f96-1b83-4636-9a7d-57a5892e5e2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:05.769721 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769253    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-host-cni-bin\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.769721 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769288    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-host-run-multus-certs\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.769721 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769293    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a1db6ec-8b9d-4f0a-8bf0-de72e290cdd1-hosts-file\") pod \"node-resolver-2kswj\" (UID: \"1a1db6ec-8b9d-4f0a-8bf0-de72e290cdd1\") " pod="openshift-dns/node-resolver-2kswj"
Sep 10 04:41:05.770463 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769311    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a75e8f96-1b83-4636-9a7d-57a5892e5e2b-etc-selinux\") pod \"aws-ebs-csi-driver-node-b78sr\" (UID: \"a75e8f96-1b83-4636-9a7d-57a5892e5e2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:05.770463 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769349    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a1db6ec-8b9d-4f0a-8bf0-de72e290cdd1-hosts-file\") pod \"node-resolver-2kswj\" (UID: \"1a1db6ec-8b9d-4f0a-8bf0-de72e290cdd1\") " pod="openshift-dns/node-resolver-2kswj"
Sep 10 04:41:05.770463 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769341    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-host-var-lib-kubelet\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.770463 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769376    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-host-var-lib-kubelet\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.770463 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769386    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a75e8f96-1b83-4636-9a7d-57a5892e5e2b-device-dir\") pod \"aws-ebs-csi-driver-node-b78sr\" (UID: \"a75e8f96-1b83-4636-9a7d-57a5892e5e2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:05.770463 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dafe7236-9f57-4f88-97d5-b30b2f8ee269-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ms5c5\" (UID: \"dafe7236-9f57-4f88-97d5-b30b2f8ee269\") " pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.770463 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769441    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-var-lib-openvswitch\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.770463 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769414    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 04:41:05.770463 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769470    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret\") pod \"global-pull-secret-syncer-pclmn\" (UID: \"920551bb-5e01-45e3-80cd-87f46761fb29\") " pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:05.770463 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769485    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-var-lib-openvswitch\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.770463 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769493    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-etc-modprobe-d\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.770463 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769527    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-host\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.770463 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769550    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dafe7236-9f57-4f88-97d5-b30b2f8ee269-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ms5c5\" (UID: \"dafe7236-9f57-4f88-97d5-b30b2f8ee269\") " pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.770463 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769552    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6qcw\" (UniqueName: \"kubernetes.io/projected/c4342bc6-a78e-4341-bdd6-c9e991c1c131-kube-api-access-b6qcw\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.770463 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769568    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7fc6a301-6f28-4a4f-b547-2991c1cbb8b6-konnectivity-ca\") pod \"konnectivity-agent-c69jd\" (UID: \"7fc6a301-6f28-4a4f-b547-2991c1cbb8b6\") " pod="kube-system/konnectivity-agent-c69jd"
Sep 10 04:41:05.770463 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769582    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-etc-modprobe-d\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.770463 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769590    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8fa349bb-d0f7-48c7-ba38-d533479eb889-cni-binary-copy\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.770463 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769615    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-host\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.771280 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769620    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/920551bb-5e01-45e3-80cd-87f46761fb29-kubelet-config\") pod \"global-pull-secret-syncer-pclmn\" (UID: \"920551bb-5e01-45e3-80cd-87f46761fb29\") " pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:05.771280 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769644    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-system-cni-dir\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.771280 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769453    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a75e8f96-1b83-4636-9a7d-57a5892e5e2b-device-dir\") pod \"aws-ebs-csi-driver-node-b78sr\" (UID: \"a75e8f96-1b83-4636-9a7d-57a5892e5e2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:05.771280 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769670    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-host-run-k8s-cni-cncf-io\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.771280 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:05.769697    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 04:41:05.771280 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769710    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-host-run-netns\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.771280 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769744    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-system-cni-dir\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.771280 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769749    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7hpx\" (UniqueName: \"kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx\") pod \"network-check-target-47tk5\" (UID: \"7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8\") " pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:05.771280 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:05.769767    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret podName:920551bb-5e01-45e3-80cd-87f46761fb29 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:06.269746658 +0000 UTC m=+3.031515841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret") pod "global-pull-secret-syncer-pclmn" (UID: "920551bb-5e01-45e3-80cd-87f46761fb29") : object "kube-system"/"original-pull-secret" not registered
Sep 10 04:41:05.771280 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769798    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/920551bb-5e01-45e3-80cd-87f46761fb29-kubelet-config\") pod \"global-pull-secret-syncer-pclmn\" (UID: \"920551bb-5e01-45e3-80cd-87f46761fb29\") " pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:05.771280 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769815    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-host-run-k8s-cni-cncf-io\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.771280 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769853    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47240839-28f9-4291-b91f-b22c142ce1f0-host-slash\") pod \"iptables-alerter-wg9t7\" (UID: \"47240839-28f9-4291-b91f-b22c142ce1f0\") " pod="openshift-network-operator/iptables-alerter-wg9t7"
Sep 10 04:41:05.771280 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769871    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-host-run-netns\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.771280 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769888    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a75e8f96-1b83-4636-9a7d-57a5892e5e2b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b78sr\" (UID: \"a75e8f96-1b83-4636-9a7d-57a5892e5e2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:05.771280 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769918    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-systemd-units\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.771280 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769925    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47240839-28f9-4291-b91f-b22c142ce1f0-host-slash\") pod \"iptables-alerter-wg9t7\" (UID: \"47240839-28f9-4291-b91f-b22c142ce1f0\") " pod="openshift-network-operator/iptables-alerter-wg9t7"
Sep 10 04:41:05.771280 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769943    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs\") pod \"network-metrics-daemon-7krfk\" (UID: \"4bbb6665-dfae-48eb-9f23-f5147f2ccebb\") " pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:05.772190 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769987    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-systemd-units\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.772190 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.769972    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a75e8f96-1b83-4636-9a7d-57a5892e5e2b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b78sr\" (UID: \"a75e8f96-1b83-4636-9a7d-57a5892e5e2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:05.772190 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:05.770020    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 04:41:05.772190 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770041    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-var-lib-kubelet\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.772190 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:05.770086    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs podName:4bbb6665-dfae-48eb-9f23-f5147f2ccebb nodeName:}" failed. No retries permitted until 2026-09-10 04:41:06.270053799 +0000 UTC m=+3.031822973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs") pod "network-metrics-daemon-7krfk" (UID: "4bbb6665-dfae-48eb-9f23-f5147f2ccebb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 04:41:05.772190 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770104    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85bjj\" (UniqueName: \"kubernetes.io/projected/47240839-28f9-4291-b91f-b22c142ce1f0-kube-api-access-85bjj\") pod \"iptables-alerter-wg9t7\" (UID: \"47240839-28f9-4291-b91f-b22c142ce1f0\") " pod="openshift-network-operator/iptables-alerter-wg9t7"
Sep 10 04:41:05.772190 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770113    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-var-lib-kubelet\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.772190 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770131    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9d909143-4c70-4fe1-900a-3cd520db32e5-ovnkube-config\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.772190 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770224    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8fa349bb-d0f7-48c7-ba38-d533479eb889-cni-binary-copy\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.772190 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770261    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a75e8f96-1b83-4636-9a7d-57a5892e5e2b-registration-dir\") pod \"aws-ebs-csi-driver-node-b78sr\" (UID: \"a75e8f96-1b83-4636-9a7d-57a5892e5e2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:05.772190 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770286    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-host-run-netns\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.772190 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770321    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-node-log\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.772190 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770342    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a75e8f96-1b83-4636-9a7d-57a5892e5e2b-registration-dir\") pod \"aws-ebs-csi-driver-node-b78sr\" (UID: \"a75e8f96-1b83-4636-9a7d-57a5892e5e2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:05.772190 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770374    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-host-run-netns\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.772190 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770383    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-node-log\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.772190 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770345    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-host-run-ovn-kubernetes\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.772190 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770417    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-etc-systemd\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.773132 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770423    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-host-run-ovn-kubernetes\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.773132 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770441    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02eb37ae-67ce-4454-b34a-e3666a73f999-host\") pod \"node-ca-n2zlm\" (UID: \"02eb37ae-67ce-4454-b34a-e3666a73f999\") " pod="openshift-image-registry/node-ca-n2zlm"
Sep 10 04:41:05.773132 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770467    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5n6v\" (UniqueName: \"kubernetes.io/projected/02eb37ae-67ce-4454-b34a-e3666a73f999-kube-api-access-p5n6v\") pod \"node-ca-n2zlm\" (UID: \"02eb37ae-67ce-4454-b34a-e3666a73f999\") " pod="openshift-image-registry/node-ca-n2zlm"
Sep 10 04:41:05.773132 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770494    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k74vc\" (UniqueName: \"kubernetes.io/projected/1a1db6ec-8b9d-4f0a-8bf0-de72e290cdd1-kube-api-access-k74vc\") pod \"node-resolver-2kswj\" (UID: \"1a1db6ec-8b9d-4f0a-8bf0-de72e290cdd1\") " pod="openshift-dns/node-resolver-2kswj"
Sep 10 04:41:05.773132 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770497    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02eb37ae-67ce-4454-b34a-e3666a73f999-host\") pod \"node-ca-n2zlm\" (UID: \"02eb37ae-67ce-4454-b34a-e3666a73f999\") " pod="openshift-image-registry/node-ca-n2zlm"
Sep 10 04:41:05.773132 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770520    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-host-kubelet\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.773132 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770543    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4342bc6-a78e-4341-bdd6-c9e991c1c131-tmp\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.773132 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770565    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9d909143-4c70-4fe1-900a-3cd520db32e5-ovnkube-config\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.773132 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770591    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpph5\" (UniqueName: \"kubernetes.io/projected/a75e8f96-1b83-4636-9a7d-57a5892e5e2b-kube-api-access-lpph5\") pod \"aws-ebs-csi-driver-node-b78sr\" (UID: \"a75e8f96-1b83-4636-9a7d-57a5892e5e2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:05.773132 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770617    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dafe7236-9f57-4f88-97d5-b30b2f8ee269-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ms5c5\" (UID: \"dafe7236-9f57-4f88-97d5-b30b2f8ee269\") " pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.773132 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770651    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-host-slash\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.773132 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770674    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-run-ovn\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.773132 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770468    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-etc-systemd\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.773132 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770698    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.773132 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770725    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g96d4\" (UniqueName: \"kubernetes.io/projected/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-kube-api-access-g96d4\") pod \"network-metrics-daemon-7krfk\" (UID: \"4bbb6665-dfae-48eb-9f23-f5147f2ccebb\") " pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:05.773132 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770740    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-host-kubelet\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.773132 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770752    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-multus-conf-dir\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.773944 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770780    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-run-ovn\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.773944 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770789    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-etc-kubernetes\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.773944 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770823    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dafe7236-9f57-4f88-97d5-b30b2f8ee269-system-cni-dir\") pod \"multus-additional-cni-plugins-ms5c5\" (UID: \"dafe7236-9f57-4f88-97d5-b30b2f8ee269\") " pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.773944 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770849    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dafe7236-9f57-4f88-97d5-b30b2f8ee269-cnibin\") pod \"multus-additional-cni-plugins-ms5c5\" (UID: \"dafe7236-9f57-4f88-97d5-b30b2f8ee269\") " pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.773944 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770875    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrg95\" (UniqueName: \"kubernetes.io/projected/9d909143-4c70-4fe1-900a-3cd520db32e5-kube-api-access-jrg95\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.773944 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770899    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/920551bb-5e01-45e3-80cd-87f46761fb29-dbus\") pod \"global-pull-secret-syncer-pclmn\" (UID: \"920551bb-5e01-45e3-80cd-87f46761fb29\") " pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:05.773944 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770934    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-sys\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.773944 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770956    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-cnibin\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.773944 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.770979    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb4s5\" (UniqueName: \"kubernetes.io/projected/8fa349bb-d0f7-48c7-ba38-d533479eb889-kube-api-access-rb4s5\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.773944 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771000    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7fc6a301-6f28-4a4f-b547-2991c1cbb8b6-agent-certs\") pod \"konnectivity-agent-c69jd\" (UID: \"7fc6a301-6f28-4a4f-b547-2991c1cbb8b6\") " pod="kube-system/konnectivity-agent-c69jd"
Sep 10 04:41:05.773944 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771034    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-multus-socket-dir-parent\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.773944 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771043    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-host-slash\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.773944 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771058    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dafe7236-9f57-4f88-97d5-b30b2f8ee269-cni-binary-copy\") pod \"multus-additional-cni-plugins-ms5c5\" (UID: \"dafe7236-9f57-4f88-97d5-b30b2f8ee269\") " pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.773944 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771121    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dafe7236-9f57-4f88-97d5-b30b2f8ee269-cnibin\") pod \"multus-additional-cni-plugins-ms5c5\" (UID: \"dafe7236-9f57-4f88-97d5-b30b2f8ee269\") " pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.773944 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771139    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dafe7236-9f57-4f88-97d5-b30b2f8ee269-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ms5c5\" (UID: \"dafe7236-9f57-4f88-97d5-b30b2f8ee269\") " pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.773944 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771170    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-multus-conf-dir\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.773944 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771168    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-run-systemd\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.773944 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771215    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-etc-kubernetes\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.774862 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771225    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-run-systemd\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.774862 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771240    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-etc-sysctl-d\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.774862 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771265    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/47240839-28f9-4291-b91f-b22c142ce1f0-iptables-alerter-script\") pod \"iptables-alerter-wg9t7\" (UID: \"47240839-28f9-4291-b91f-b22c142ce1f0\") " pod="openshift-network-operator/iptables-alerter-wg9t7"
Sep 10 04:41:05.774862 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771316    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m69s\" (UniqueName: \"kubernetes.io/projected/dafe7236-9f57-4f88-97d5-b30b2f8ee269-kube-api-access-9m69s\") pod \"multus-additional-cni-plugins-ms5c5\" (UID: \"dafe7236-9f57-4f88-97d5-b30b2f8ee269\") " pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.774862 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771351    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.774862 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771379    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02eb37ae-67ce-4454-b34a-e3666a73f999-serviceca\") pod \"node-ca-n2zlm\" (UID: \"02eb37ae-67ce-4454-b34a-e3666a73f999\") " pod="openshift-image-registry/node-ca-n2zlm"
Sep 10 04:41:05.774862 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771440    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-host-var-lib-cni-bin\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.774862 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771468    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a75e8f96-1b83-4636-9a7d-57a5892e5e2b-socket-dir\") pod \"aws-ebs-csi-driver-node-b78sr\" (UID: \"a75e8f96-1b83-4636-9a7d-57a5892e5e2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:05.774862 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771536    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a75e8f96-1b83-4636-9a7d-57a5892e5e2b-sys-fs\") pod \"aws-ebs-csi-driver-node-b78sr\" (UID: \"a75e8f96-1b83-4636-9a7d-57a5892e5e2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:05.774862 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771597    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-etc-openvswitch\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.774862 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771617    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a75e8f96-1b83-4636-9a7d-57a5892e5e2b-socket-dir\") pod \"aws-ebs-csi-driver-node-b78sr\" (UID: \"a75e8f96-1b83-4636-9a7d-57a5892e5e2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:05.774862 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771622    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-log-socket\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.774862 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771667    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-host-cni-netd\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.774862 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771730    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9d909143-4c70-4fe1-900a-3cd520db32e5-ovnkube-script-lib\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.774862 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771756    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8fa349bb-d0f7-48c7-ba38-d533479eb889-multus-daemon-config\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.774862 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771795    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/920551bb-5e01-45e3-80cd-87f46761fb29-dbus\") pod \"global-pull-secret-syncer-pclmn\" (UID: \"920551bb-5e01-45e3-80cd-87f46761fb29\") " pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:05.774862 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771904    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-cnibin\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.775489 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.771980    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-etc-openvswitch\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.775489 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.772343    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8fa349bb-d0f7-48c7-ba38-d533479eb889-multus-daemon-config\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.775489 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.772409    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-sys\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.775489 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.772458    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-log-socket\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.775489 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.772523    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9d909143-4c70-4fe1-900a-3cd520db32e5-host-cni-netd\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.775489 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.772712    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dafe7236-9f57-4f88-97d5-b30b2f8ee269-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ms5c5\" (UID: \"dafe7236-9f57-4f88-97d5-b30b2f8ee269\") " pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.775489 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.772789    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a75e8f96-1b83-4636-9a7d-57a5892e5e2b-sys-fs\") pod \"aws-ebs-csi-driver-node-b78sr\" (UID: \"a75e8f96-1b83-4636-9a7d-57a5892e5e2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:05.775489 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.773056    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9d909143-4c70-4fe1-900a-3cd520db32e5-ovnkube-script-lib\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.775489 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.773493    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02eb37ae-67ce-4454-b34a-e3666a73f999-serviceca\") pod \"node-ca-n2zlm\" (UID: \"02eb37ae-67ce-4454-b34a-e3666a73f999\") " pod="openshift-image-registry/node-ca-n2zlm"
Sep 10 04:41:05.775489 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.773758    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/47240839-28f9-4291-b91f-b22c142ce1f0-iptables-alerter-script\") pod \"iptables-alerter-wg9t7\" (UID: \"47240839-28f9-4291-b91f-b22c142ce1f0\") " pod="openshift-network-operator/iptables-alerter-wg9t7"
Sep 10 04:41:05.775489 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.773803    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-host-var-lib-cni-bin\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.775489 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.773833    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dafe7236-9f57-4f88-97d5-b30b2f8ee269-system-cni-dir\") pod \"multus-additional-cni-plugins-ms5c5\" (UID: \"dafe7236-9f57-4f88-97d5-b30b2f8ee269\") " pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.775489 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.773868    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8fa349bb-d0f7-48c7-ba38-d533479eb889-multus-socket-dir-parent\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.775489 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.773924    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4342bc6-a78e-4341-bdd6-c9e991c1c131-etc-sysctl-d\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.775489 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.774448    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dafe7236-9f57-4f88-97d5-b30b2f8ee269-cni-binary-copy\") pod \"multus-additional-cni-plugins-ms5c5\" (UID: \"dafe7236-9f57-4f88-97d5-b30b2f8ee269\") " pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.775489 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.774488    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4342bc6-a78e-4341-bdd6-c9e991c1c131-etc-tuned\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.775489 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.774557    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7fc6a301-6f28-4a4f-b547-2991c1cbb8b6-agent-certs\") pod \"konnectivity-agent-c69jd\" (UID: \"7fc6a301-6f28-4a4f-b547-2991c1cbb8b6\") " pod="kube-system/konnectivity-agent-c69jd"
Sep 10 04:41:05.776055 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.774674    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dafe7236-9f57-4f88-97d5-b30b2f8ee269-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ms5c5\" (UID: \"dafe7236-9f57-4f88-97d5-b30b2f8ee269\") " pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.776055 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.774727    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4342bc6-a78e-4341-bdd6-c9e991c1c131-tmp\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.776055 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.775024    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d909143-4c70-4fe1-900a-3cd520db32e5-ovn-node-metrics-cert\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.780647 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:05.780420    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 04:41:05.780647 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:05.780458    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 04:41:05.780647 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:05.780474    2575 projected.go:196] Error preparing data for projected volume kube-api-access-r7hpx for pod openshift-network-diagnostics/network-check-target-47tk5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 04:41:05.780647 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:05.780540    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx podName:7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:06.280520398 +0000 UTC m=+3.042289572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r7hpx" (UniqueName: "kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx") pod "network-check-target-47tk5" (UID: "7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 04:41:05.786899 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.784087    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6qcw\" (UniqueName: \"kubernetes.io/projected/c4342bc6-a78e-4341-bdd6-c9e991c1c131-kube-api-access-b6qcw\") pod \"tuned-psjfv\" (UID: \"c4342bc6-a78e-4341-bdd6-c9e991c1c131\") " pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:05.786899 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.784997    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrg95\" (UniqueName: \"kubernetes.io/projected/9d909143-4c70-4fe1-900a-3cd520db32e5-kube-api-access-jrg95\") pod \"ovnkube-node-j2hd2\" (UID: \"9d909143-4c70-4fe1-900a-3cd520db32e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.786899 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.785792    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5n6v\" (UniqueName: \"kubernetes.io/projected/02eb37ae-67ce-4454-b34a-e3666a73f999-kube-api-access-p5n6v\") pod \"node-ca-n2zlm\" (UID: \"02eb37ae-67ce-4454-b34a-e3666a73f999\") " pod="openshift-image-registry/node-ca-n2zlm"
Sep 10 04:41:05.786899 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.785982    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85bjj\" (UniqueName: \"kubernetes.io/projected/47240839-28f9-4291-b91f-b22c142ce1f0-kube-api-access-85bjj\") pod \"iptables-alerter-wg9t7\" (UID: \"47240839-28f9-4291-b91f-b22c142ce1f0\") " pod="openshift-network-operator/iptables-alerter-wg9t7"
Sep 10 04:41:05.786899 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.786719    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g96d4\" (UniqueName: \"kubernetes.io/projected/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-kube-api-access-g96d4\") pod \"network-metrics-daemon-7krfk\" (UID: \"4bbb6665-dfae-48eb-9f23-f5147f2ccebb\") " pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:05.786899 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.786719    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpph5\" (UniqueName: \"kubernetes.io/projected/a75e8f96-1b83-4636-9a7d-57a5892e5e2b-kube-api-access-lpph5\") pod \"aws-ebs-csi-driver-node-b78sr\" (UID: \"a75e8f96-1b83-4636-9a7d-57a5892e5e2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:05.786899 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.786848    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb4s5\" (UniqueName: \"kubernetes.io/projected/8fa349bb-d0f7-48c7-ba38-d533479eb889-kube-api-access-rb4s5\") pod \"multus-8r87c\" (UID: \"8fa349bb-d0f7-48c7-ba38-d533479eb889\") " pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.787489 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.787468    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m69s\" (UniqueName: \"kubernetes.io/projected/dafe7236-9f57-4f88-97d5-b30b2f8ee269-kube-api-access-9m69s\") pod \"multus-additional-cni-plugins-ms5c5\" (UID: \"dafe7236-9f57-4f88-97d5-b30b2f8ee269\") " pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.787993 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.787963    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k74vc\" (UniqueName: \"kubernetes.io/projected/1a1db6ec-8b9d-4f0a-8bf0-de72e290cdd1-kube-api-access-k74vc\") pod \"node-resolver-2kswj\" (UID: \"1a1db6ec-8b9d-4f0a-8bf0-de72e290cdd1\") " pod="openshift-dns/node-resolver-2kswj"
Sep 10 04:41:05.800574 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.800534    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 04:36:04 +0000 UTC" deadline="2028-06-12 09:26:55.56829511 +0000 UTC"
Sep 10 04:41:05.800574 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.800575    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15388h45m49.767723883s"
Sep 10 04:41:05.951744 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.951663    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c69jd"
Sep 10 04:41:05.961791 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.961765    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2kswj"
Sep 10 04:41:05.970533 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.970508    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ms5c5"
Sep 10 04:41:05.975192 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.975170    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8r87c"
Sep 10 04:41:05.984631 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.984611    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wg9t7"
Sep 10 04:41:05.992196 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.992175    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:05.999736 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:05.999717    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr"
Sep 10 04:41:06.008373 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:06.008345    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-psjfv"
Sep 10 04:41:06.013970 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:06.013945    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n2zlm"
Sep 10 04:41:06.175464 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:06.175440    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 04:41:06.276159 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:06.276086    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret\") pod \"global-pull-secret-syncer-pclmn\" (UID: \"920551bb-5e01-45e3-80cd-87f46761fb29\") " pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:06.276159 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:06.276130    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs\") pod \"network-metrics-daemon-7krfk\" (UID: \"4bbb6665-dfae-48eb-9f23-f5147f2ccebb\") " pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:06.276331 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:06.276225    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 04:41:06.276331 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:06.276280    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs podName:4bbb6665-dfae-48eb-9f23-f5147f2ccebb nodeName:}" failed. No retries permitted until 2026-09-10 04:41:07.276262738 +0000 UTC m=+4.038031898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs") pod "network-metrics-daemon-7krfk" (UID: "4bbb6665-dfae-48eb-9f23-f5147f2ccebb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 04:41:06.276331 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:06.276224    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 04:41:06.276432 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:06.276371    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret podName:920551bb-5e01-45e3-80cd-87f46761fb29 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:07.276355637 +0000 UTC m=+4.038124802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret") pod "global-pull-secret-syncer-pclmn" (UID: "920551bb-5e01-45e3-80cd-87f46761fb29") : object "kube-system"/"original-pull-secret" not registered
Sep 10 04:41:06.377324 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:06.377283    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7hpx\" (UniqueName: \"kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx\") pod \"network-check-target-47tk5\" (UID: \"7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8\") " pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:06.377456 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:06.377435    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 04:41:06.377506 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:06.377456    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 04:41:06.377506 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:06.377466    2575 projected.go:196] Error preparing data for projected volume kube-api-access-r7hpx for pod openshift-network-diagnostics/network-check-target-47tk5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 04:41:06.377575 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:06.377517    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx podName:7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:07.37750135 +0000 UTC m=+4.139270510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r7hpx" (UniqueName: "kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx") pod "network-check-target-47tk5" (UID: "7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 04:41:06.732055 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:06.731861    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1db6ec_8b9d_4f0a_8bf0_de72e290cdd1.slice/crio-bb12f124f8a3685beac9dc1aaf6f5b9f61f8295fbab3ac177ff945eef79fccaf WatchSource:0}: Error finding container bb12f124f8a3685beac9dc1aaf6f5b9f61f8295fbab3ac177ff945eef79fccaf: Status 404 returned error can't find the container with id bb12f124f8a3685beac9dc1aaf6f5b9f61f8295fbab3ac177ff945eef79fccaf
Sep 10 04:41:06.732437 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:06.732399    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc6a301_6f28_4a4f_b547_2991c1cbb8b6.slice/crio-37b78fc6ebdc9122e679d391f1c41dd6fa42184a2be25ba9e22377a0ae9ea270 WatchSource:0}: Error finding container 37b78fc6ebdc9122e679d391f1c41dd6fa42184a2be25ba9e22377a0ae9ea270: Status 404 returned error can't find the container with id 37b78fc6ebdc9122e679d391f1c41dd6fa42184a2be25ba9e22377a0ae9ea270
Sep 10 04:41:06.733495 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:06.733468    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d909143_4c70_4fe1_900a_3cd520db32e5.slice/crio-358d21720f2b0eb6ac0945fa2dcc50140ad8f58f2ed7ffdbd2491358fc0bf3b2 WatchSource:0}: Error finding container 358d21720f2b0eb6ac0945fa2dcc50140ad8f58f2ed7ffdbd2491358fc0bf3b2: Status 404 returned error can't find the container with id 358d21720f2b0eb6ac0945fa2dcc50140ad8f58f2ed7ffdbd2491358fc0bf3b2
Sep 10 04:41:06.735260 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:06.734875    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa349bb_d0f7_48c7_ba38_d533479eb889.slice/crio-25c99ccfd932db18fb447454cf359eb10683a4173d68fa485383022443561f96 WatchSource:0}: Error finding container 25c99ccfd932db18fb447454cf359eb10683a4173d68fa485383022443561f96: Status 404 returned error can't find the container with id 25c99ccfd932db18fb447454cf359eb10683a4173d68fa485383022443561f96
Sep 10 04:41:06.738353 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:06.738327    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47240839_28f9_4291_b91f_b22c142ce1f0.slice/crio-5c7783235ab0992451586ea73515647ffe109578ee308c95ad9cd399eb201cb4 WatchSource:0}: Error finding container 5c7783235ab0992451586ea73515647ffe109578ee308c95ad9cd399eb201cb4: Status 404 returned error can't find the container with id 5c7783235ab0992451586ea73515647ffe109578ee308c95ad9cd399eb201cb4
Sep 10 04:41:06.740143 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:06.739595    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75e8f96_1b83_4636_9a7d_57a5892e5e2b.slice/crio-b1cdd8515adf0ef47e5cf6a450fb41e776415a188f41d3aff8ba937bd670bdf2 WatchSource:0}: Error finding container b1cdd8515adf0ef47e5cf6a450fb41e776415a188f41d3aff8ba937bd670bdf2: Status 404 returned error can't find the container with id b1cdd8515adf0ef47e5cf6a450fb41e776415a188f41d3aff8ba937bd670bdf2
Sep 10 04:41:06.740481 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:06.740371    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddafe7236_9f57_4f88_97d5_b30b2f8ee269.slice/crio-70692c7928fa394482bfcf4734c3ed382c0124f575b51ca127b8fef4e50073aa WatchSource:0}: Error finding container 70692c7928fa394482bfcf4734c3ed382c0124f575b51ca127b8fef4e50073aa: Status 404 returned error can't find the container with id 70692c7928fa394482bfcf4734c3ed382c0124f575b51ca127b8fef4e50073aa
Sep 10 04:41:06.743102 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:06.742701    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02eb37ae_67ce_4454_b34a_e3666a73f999.slice/crio-39f7a3a141956a393ffa5b493be99efbe8e286ab47d7ee555782b89258e6805e WatchSource:0}: Error finding container 39f7a3a141956a393ffa5b493be99efbe8e286ab47d7ee555782b89258e6805e: Status 404 returned error can't find the container with id 39f7a3a141956a393ffa5b493be99efbe8e286ab47d7ee555782b89258e6805e
Sep 10 04:41:06.744239 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:06.744215    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4342bc6_a78e_4341_bdd6_c9e991c1c131.slice/crio-df937034509cafe88c21026418343273a8ffef9d1154ff88c00390c6c26e1741 WatchSource:0}: Error finding container df937034509cafe88c21026418343273a8ffef9d1154ff88c00390c6c26e1741: Status 404 returned error can't find the container with id df937034509cafe88c21026418343273a8ffef9d1154ff88c00390c6c26e1741
Sep 10 04:41:06.801695 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:06.801655    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 04:36:04 +0000 UTC" deadline="2028-02-14 08:40:15.167169729 +0000 UTC"
Sep 10 04:41:06.801798 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:06.801694    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12531h59m8.365479592s"
Sep 10 04:41:06.865904 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:06.865847    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-70.ec2.internal" event={"ID":"9d5f561d6e275e920a4621f246ba503e","Type":"ContainerStarted","Data":"839e5ab9a3c44158812ca4480c8158f1196d9d60322e825e8374e0526436585c"}
Sep 10 04:41:06.867121 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:06.867095    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n2zlm" event={"ID":"02eb37ae-67ce-4454-b34a-e3666a73f999","Type":"ContainerStarted","Data":"39f7a3a141956a393ffa5b493be99efbe8e286ab47d7ee555782b89258e6805e"}
Sep 10 04:41:06.868262 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:06.868240    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr" event={"ID":"a75e8f96-1b83-4636-9a7d-57a5892e5e2b","Type":"ContainerStarted","Data":"b1cdd8515adf0ef47e5cf6a450fb41e776415a188f41d3aff8ba937bd670bdf2"}
Sep 10 04:41:06.869471 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:06.869442    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ms5c5" event={"ID":"dafe7236-9f57-4f88-97d5-b30b2f8ee269","Type":"ContainerStarted","Data":"70692c7928fa394482bfcf4734c3ed382c0124f575b51ca127b8fef4e50073aa"}
Sep 10 04:41:06.870615 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:06.870596    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c69jd" event={"ID":"7fc6a301-6f28-4a4f-b547-2991c1cbb8b6","Type":"ContainerStarted","Data":"37b78fc6ebdc9122e679d391f1c41dd6fa42184a2be25ba9e22377a0ae9ea270"}
Sep 10 04:41:06.871746 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:06.871725    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2" event={"ID":"9d909143-4c70-4fe1-900a-3cd520db32e5","Type":"ContainerStarted","Data":"358d21720f2b0eb6ac0945fa2dcc50140ad8f58f2ed7ffdbd2491358fc0bf3b2"}
Sep 10 04:41:06.872868 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:06.872848    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2kswj" event={"ID":"1a1db6ec-8b9d-4f0a-8bf0-de72e290cdd1","Type":"ContainerStarted","Data":"bb12f124f8a3685beac9dc1aaf6f5b9f61f8295fbab3ac177ff945eef79fccaf"}
Sep 10 04:41:06.874215 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:06.874191    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-psjfv" event={"ID":"c4342bc6-a78e-4341-bdd6-c9e991c1c131","Type":"ContainerStarted","Data":"df937034509cafe88c21026418343273a8ffef9d1154ff88c00390c6c26e1741"}
Sep 10 04:41:06.875333 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:06.875309    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wg9t7" event={"ID":"47240839-28f9-4291-b91f-b22c142ce1f0","Type":"ContainerStarted","Data":"5c7783235ab0992451586ea73515647ffe109578ee308c95ad9cd399eb201cb4"}
Sep 10 04:41:06.876310 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:06.876289    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8r87c" event={"ID":"8fa349bb-d0f7-48c7-ba38-d533479eb889","Type":"ContainerStarted","Data":"25c99ccfd932db18fb447454cf359eb10683a4173d68fa485383022443561f96"}
Sep 10 04:41:06.880387 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:06.880348    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-70.ec2.internal" podStartSLOduration=2.880336764 podStartE2EDuration="2.880336764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:06.879848635 +0000 UTC m=+3.641617819" watchObservedRunningTime="2026-09-10 04:41:06.880336764 +0000 UTC m=+3.642105945"
Sep 10 04:41:07.283435 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:07.283334    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret\") pod \"global-pull-secret-syncer-pclmn\" (UID: \"920551bb-5e01-45e3-80cd-87f46761fb29\") " pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:07.283435 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:07.283408    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs\") pod \"network-metrics-daemon-7krfk\" (UID: \"4bbb6665-dfae-48eb-9f23-f5147f2ccebb\") " pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:07.283633 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:07.283577    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 04:41:07.283689 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:07.283642    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs podName:4bbb6665-dfae-48eb-9f23-f5147f2ccebb nodeName:}" failed. No retries permitted until 2026-09-10 04:41:09.283623853 +0000 UTC m=+6.045393038 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs") pod "network-metrics-daemon-7krfk" (UID: "4bbb6665-dfae-48eb-9f23-f5147f2ccebb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 04:41:07.284129 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:07.284060    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 04:41:07.284232 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:07.284215    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret podName:920551bb-5e01-45e3-80cd-87f46761fb29 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:09.284199129 +0000 UTC m=+6.045968291 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret") pod "global-pull-secret-syncer-pclmn" (UID: "920551bb-5e01-45e3-80cd-87f46761fb29") : object "kube-system"/"original-pull-secret" not registered
Sep 10 04:41:07.384707 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:07.384669    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7hpx\" (UniqueName: \"kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx\") pod \"network-check-target-47tk5\" (UID: \"7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8\") " pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:07.384904 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:07.384887    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 04:41:07.384970 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:07.384915    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 04:41:07.384970 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:07.384927    2575 projected.go:196] Error preparing data for projected volume kube-api-access-r7hpx for pod openshift-network-diagnostics/network-check-target-47tk5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 04:41:07.385111 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:07.384985    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx podName:7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:09.384966892 +0000 UTC m=+6.146736085 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r7hpx" (UniqueName: "kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx") pod "network-check-target-47tk5" (UID: "7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 04:41:07.571167 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:07.571089    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 04:41:07.858137 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:07.857465    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:07.858137 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:07.857484    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:07.858137 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:07.857611    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7krfk" podUID="4bbb6665-dfae-48eb-9f23-f5147f2ccebb"
Sep 10 04:41:07.858137 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:07.858008    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclmn" podUID="920551bb-5e01-45e3-80cd-87f46761fb29"
Sep 10 04:41:07.861638 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:07.861471    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:07.861638 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:07.861598    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47tk5" podUID="7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8"
Sep 10 04:41:07.898095 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:07.897861    2575 generic.go:356] "Generic (PLEG): container finished" podID="27bd344442d49c85b9c1f8743a1a1f9f" containerID="cc71d9f13abf9b56ec646e0d44df0c682bfadb24e92710276f52f21e4ee01fa1" exitCode=0
Sep 10 04:41:07.898450 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:07.898034    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-70.ec2.internal" event={"ID":"27bd344442d49c85b9c1f8743a1a1f9f","Type":"ContainerDied","Data":"cc71d9f13abf9b56ec646e0d44df0c682bfadb24e92710276f52f21e4ee01fa1"}
Sep 10 04:41:08.921692 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:08.921649    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-70.ec2.internal" event={"ID":"27bd344442d49c85b9c1f8743a1a1f9f","Type":"ContainerStarted","Data":"085685b12a7828d8f5bd573a17b906b8e3be9d0a64f33f9dfee25f928c95914f"}
Sep 10 04:41:08.939249 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:08.938087    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-70.ec2.internal" podStartSLOduration=4.938050186 podStartE2EDuration="4.938050186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:08.937939793 +0000 UTC m=+5.699708977" watchObservedRunningTime="2026-09-10 04:41:08.938050186 +0000 UTC m=+5.699819358"
Sep 10 04:41:09.300056 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:09.299839    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret\") pod \"global-pull-secret-syncer-pclmn\" (UID: \"920551bb-5e01-45e3-80cd-87f46761fb29\") " pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:09.300056 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:09.299930    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs\") pod \"network-metrics-daemon-7krfk\" (UID: \"4bbb6665-dfae-48eb-9f23-f5147f2ccebb\") " pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:09.300298 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:09.300097    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 04:41:09.300298 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:09.300176    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs podName:4bbb6665-dfae-48eb-9f23-f5147f2ccebb nodeName:}" failed. No retries permitted until 2026-09-10 04:41:13.300154838 +0000 UTC m=+10.061924024 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs") pod "network-metrics-daemon-7krfk" (UID: "4bbb6665-dfae-48eb-9f23-f5147f2ccebb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 04:41:09.301328 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:09.300672    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 04:41:09.301328 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:09.300730    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret podName:920551bb-5e01-45e3-80cd-87f46761fb29 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:13.300713993 +0000 UTC m=+10.062483167 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret") pod "global-pull-secret-syncer-pclmn" (UID: "920551bb-5e01-45e3-80cd-87f46761fb29") : object "kube-system"/"original-pull-secret" not registered
Sep 10 04:41:09.401592 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:09.400938    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7hpx\" (UniqueName: \"kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx\") pod \"network-check-target-47tk5\" (UID: \"7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8\") " pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:09.401592 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:09.401152    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 04:41:09.401592 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:09.401175    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 04:41:09.401592 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:09.401188    2575 projected.go:196] Error preparing data for projected volume kube-api-access-r7hpx for pod openshift-network-diagnostics/network-check-target-47tk5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 04:41:09.401592 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:09.401249    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx podName:7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:13.401229952 +0000 UTC m=+10.162999115 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r7hpx" (UniqueName: "kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx") pod "network-check-target-47tk5" (UID: "7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 04:41:09.857331 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:09.857293    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:09.857331 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:09.857325    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:09.857530 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:09.857440    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7krfk" podUID="4bbb6665-dfae-48eb-9f23-f5147f2ccebb"
Sep 10 04:41:09.857892 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:09.857861    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47tk5" podUID="7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8"
Sep 10 04:41:09.858005 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:09.857950    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:09.858076 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:09.858033    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclmn" podUID="920551bb-5e01-45e3-80cd-87f46761fb29"
Sep 10 04:41:11.857338 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:11.857308    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:11.857797 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:11.857308    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:11.857797 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:11.857443    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclmn" podUID="920551bb-5e01-45e3-80cd-87f46761fb29"
Sep 10 04:41:11.857797 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:11.857314    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:11.857797 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:11.857553    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7krfk" podUID="4bbb6665-dfae-48eb-9f23-f5147f2ccebb"
Sep 10 04:41:11.857797 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:11.857669    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47tk5" podUID="7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8"
Sep 10 04:41:13.335708 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:13.335671    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret\") pod \"global-pull-secret-syncer-pclmn\" (UID: \"920551bb-5e01-45e3-80cd-87f46761fb29\") " pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:13.336224 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:13.335722    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs\") pod \"network-metrics-daemon-7krfk\" (UID: \"4bbb6665-dfae-48eb-9f23-f5147f2ccebb\") " pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:13.336224 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:13.335857    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 04:41:13.336224 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:13.335870    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 04:41:13.336224 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:13.335938    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret podName:920551bb-5e01-45e3-80cd-87f46761fb29 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:21.335914919 +0000 UTC m=+18.097684102 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret") pod "global-pull-secret-syncer-pclmn" (UID: "920551bb-5e01-45e3-80cd-87f46761fb29") : object "kube-system"/"original-pull-secret" not registered
Sep 10 04:41:13.336224 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:13.335959    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs podName:4bbb6665-dfae-48eb-9f23-f5147f2ccebb nodeName:}" failed. No retries permitted until 2026-09-10 04:41:21.335948076 +0000 UTC m=+18.097717241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs") pod "network-metrics-daemon-7krfk" (UID: "4bbb6665-dfae-48eb-9f23-f5147f2ccebb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 04:41:13.436261 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:13.436225    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7hpx\" (UniqueName: \"kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx\") pod \"network-check-target-47tk5\" (UID: \"7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8\") " pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:13.436449 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:13.436385    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 04:41:13.436449 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:13.436412    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 04:41:13.436449 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:13.436425    2575 projected.go:196] Error preparing data for projected volume kube-api-access-r7hpx for pod openshift-network-diagnostics/network-check-target-47tk5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 04:41:13.436610 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:13.436492    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx podName:7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:21.43647145 +0000 UTC m=+18.198240633 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r7hpx" (UniqueName: "kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx") pod "network-check-target-47tk5" (UID: "7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 04:41:13.861178 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:13.860515    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:13.861178 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:13.860646    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclmn" podUID="920551bb-5e01-45e3-80cd-87f46761fb29"
Sep 10 04:41:13.861178 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:13.860978    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:13.861178 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:13.861094    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47tk5" podUID="7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8"
Sep 10 04:41:13.861178 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:13.861151    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:13.861451 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:13.861231    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7krfk" podUID="4bbb6665-dfae-48eb-9f23-f5147f2ccebb"
Sep 10 04:41:15.860183 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:15.860153    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:15.860636 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:15.860192    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:15.860636 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:15.860268    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:15.860636 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:15.860285    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclmn" podUID="920551bb-5e01-45e3-80cd-87f46761fb29"
Sep 10 04:41:15.860636 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:15.860369    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47tk5" podUID="7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8"
Sep 10 04:41:15.860636 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:15.860483    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7krfk" podUID="4bbb6665-dfae-48eb-9f23-f5147f2ccebb"
Sep 10 04:41:17.856725 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:17.856696    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:17.857228 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:17.856693    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:17.857228 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:17.856833    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7krfk" podUID="4bbb6665-dfae-48eb-9f23-f5147f2ccebb"
Sep 10 04:41:17.857228 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:17.856882    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:17.857228 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:17.856994    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47tk5" podUID="7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8"
Sep 10 04:41:17.857228 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:17.857107    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclmn" podUID="920551bb-5e01-45e3-80cd-87f46761fb29"
Sep 10 04:41:19.857665 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:19.857465    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:19.858152 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:19.857465    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:19.858152 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:19.857761    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7krfk" podUID="4bbb6665-dfae-48eb-9f23-f5147f2ccebb"
Sep 10 04:41:19.858152 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:19.857847    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47tk5" podUID="7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8"
Sep 10 04:41:19.858152 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:19.857498    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:19.858152 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:19.857949    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclmn" podUID="920551bb-5e01-45e3-80cd-87f46761fb29"
Sep 10 04:41:21.395799 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:21.395762    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret\") pod \"global-pull-secret-syncer-pclmn\" (UID: \"920551bb-5e01-45e3-80cd-87f46761fb29\") " pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:21.396317 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:21.395809    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs\") pod \"network-metrics-daemon-7krfk\" (UID: \"4bbb6665-dfae-48eb-9f23-f5147f2ccebb\") " pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:21.396317 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:21.395926    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 04:41:21.396317 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:21.395928    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 04:41:21.396317 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:21.395984    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs podName:4bbb6665-dfae-48eb-9f23-f5147f2ccebb nodeName:}" failed. No retries permitted until 2026-09-10 04:41:37.39596873 +0000 UTC m=+34.157737889 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs") pod "network-metrics-daemon-7krfk" (UID: "4bbb6665-dfae-48eb-9f23-f5147f2ccebb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 04:41:21.396317 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:21.395998    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret podName:920551bb-5e01-45e3-80cd-87f46761fb29 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:37.39599195 +0000 UTC m=+34.157761110 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret") pod "global-pull-secret-syncer-pclmn" (UID: "920551bb-5e01-45e3-80cd-87f46761fb29") : object "kube-system"/"original-pull-secret" not registered
Sep 10 04:41:21.496764 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:21.496724    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7hpx\" (UniqueName: \"kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx\") pod \"network-check-target-47tk5\" (UID: \"7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8\") " pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:21.496931 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:21.496870    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 04:41:21.496931 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:21.496889    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 04:41:21.496931 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:21.496899    2575 projected.go:196] Error preparing data for projected volume kube-api-access-r7hpx for pod openshift-network-diagnostics/network-check-target-47tk5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 04:41:21.497056 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:21.496955    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx podName:7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:37.496939463 +0000 UTC m=+34.258708645 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r7hpx" (UniqueName: "kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx") pod "network-check-target-47tk5" (UID: "7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 04:41:21.856699 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:21.856661    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:21.856865 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:21.856709    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:21.856865 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:21.856672    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:21.856865 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:21.856794    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47tk5" podUID="7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8"
Sep 10 04:41:21.857029 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:21.856883    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclmn" podUID="920551bb-5e01-45e3-80cd-87f46761fb29"
Sep 10 04:41:21.857029 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:21.856984    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7krfk" podUID="4bbb6665-dfae-48eb-9f23-f5147f2ccebb"
Sep 10 04:41:23.857437 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:23.857399    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:23.857855 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:23.857504    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47tk5" podUID="7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8"
Sep 10 04:41:23.857855 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:23.857584    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:23.857855 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:23.857706    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7krfk" podUID="4bbb6665-dfae-48eb-9f23-f5147f2ccebb"
Sep 10 04:41:23.857855 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:23.857752    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:23.857855 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:23.857825    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclmn" podUID="920551bb-5e01-45e3-80cd-87f46761fb29"
Sep 10 04:41:24.957464 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:24.957226    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n2zlm" event={"ID":"02eb37ae-67ce-4454-b34a-e3666a73f999","Type":"ContainerStarted","Data":"9cb7a49d8e8906153a81794bb53062573c641f82e35d7cd95a41d5170c9bf640"}
Sep 10 04:41:24.959198 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:24.959165    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr" event={"ID":"a75e8f96-1b83-4636-9a7d-57a5892e5e2b","Type":"ContainerStarted","Data":"a97e102386a106a428d49150de76e8c9f31196f6a63316dd364bb2c34373f46a"}
Sep 10 04:41:24.961652 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:24.961627    2575 generic.go:356] "Generic (PLEG): container finished" podID="dafe7236-9f57-4f88-97d5-b30b2f8ee269" containerID="0ffe210370d296c814ee97cfe7e8ebf38bf91aa1571b312b29e0497047d84793" exitCode=0
Sep 10 04:41:24.961751 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:24.961710    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ms5c5" event={"ID":"dafe7236-9f57-4f88-97d5-b30b2f8ee269","Type":"ContainerDied","Data":"0ffe210370d296c814ee97cfe7e8ebf38bf91aa1571b312b29e0497047d84793"}
Sep 10 04:41:24.963732 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:24.963705    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c69jd" event={"ID":"7fc6a301-6f28-4a4f-b547-2991c1cbb8b6","Type":"ContainerStarted","Data":"7867ab1a47cc44cd0943d16f7a5d299f1c9159a2be4dfc03f926dbf99370172f"}
Sep 10 04:41:24.968664 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:24.968635    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2hd2_9d909143-4c70-4fe1-900a-3cd520db32e5/ovn-acl-logging/0.log"
Sep 10 04:41:24.969535 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:24.969510    2575 generic.go:356] "Generic (PLEG): container finished" podID="9d909143-4c70-4fe1-900a-3cd520db32e5" containerID="c93b62ddb98fef9dc1bc673c647e2be107e519d9e552ee26a6864acfe98302c0" exitCode=1
Sep 10 04:41:24.969624 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:24.969581    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2" event={"ID":"9d909143-4c70-4fe1-900a-3cd520db32e5","Type":"ContainerStarted","Data":"f19bf837f972e4aec23cc8ceca530a4c5bae05add998aedfd80dc1ed65b4a938"}
Sep 10 04:41:24.969679 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:24.969646    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2" event={"ID":"9d909143-4c70-4fe1-900a-3cd520db32e5","Type":"ContainerStarted","Data":"9e74a42f8a16c39ea5796d3e3398b20947092bd6248ba19faba6b5eaa9d5cb8b"}
Sep 10 04:41:24.969736 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:24.969697    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2" event={"ID":"9d909143-4c70-4fe1-900a-3cd520db32e5","Type":"ContainerStarted","Data":"3fb0925bbe4740c9dd86cdc7b491d626bfc2b87c9c700cacdb655a9e9a21dc01"}
Sep 10 04:41:24.969786 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:24.969745    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2" event={"ID":"9d909143-4c70-4fe1-900a-3cd520db32e5","Type":"ContainerStarted","Data":"0a493a2dc5b5b6cdd01368c6249dd0ae6fee5d781ebdab0c32b35f9b23235fce"}
Sep 10 04:41:24.969838 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:24.969792    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2" event={"ID":"9d909143-4c70-4fe1-900a-3cd520db32e5","Type":"ContainerDied","Data":"c93b62ddb98fef9dc1bc673c647e2be107e519d9e552ee26a6864acfe98302c0"}
Sep 10 04:41:24.969893 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:24.969843    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2" event={"ID":"9d909143-4c70-4fe1-900a-3cd520db32e5","Type":"ContainerStarted","Data":"6cb89956afe5ffd97bd8584514b44eedf0e043fbe3b0dd9dc46e9077009fac8d"}
Sep 10 04:41:24.971313 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:24.971260    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2kswj" event={"ID":"1a1db6ec-8b9d-4f0a-8bf0-de72e290cdd1","Type":"ContainerStarted","Data":"bc63ff76eaa31e41fceda364a0aea3397ac9a0133c880a7d0654cd1ec9d27693"}
Sep 10 04:41:24.972517 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:24.972146    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n2zlm" podStartSLOduration=3.462285631 podStartE2EDuration="20.972133886s" totalImagesPullingTime="17.509848255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:24.971801703 +0000 UTC m=+21.733570890" watchObservedRunningTime="2026-09-10 04:41:24.972133886 +0000 UTC m=+21.733903067"
Sep 10 04:41:24.973251 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:24.973214    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-psjfv" event={"ID":"c4342bc6-a78e-4341-bdd6-c9e991c1c131","Type":"ContainerStarted","Data":"0d6c4abaa2ab1b09412558a5be2d75608ade52d19109175dac81f3c576d353a5"}
Sep 10 04:41:24.976443 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:24.976370    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8r87c" event={"ID":"8fa349bb-d0f7-48c7-ba38-d533479eb889","Type":"ContainerStarted","Data":"2289dff55e8298e8837793768284a2b28c91e1d37aea196ecb75ca83a94d34ec"}
Sep 10 04:41:24.989806 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:24.989767    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2kswj" podStartSLOduration=4.469597676 podStartE2EDuration="21.989756002s" totalImagesPullingTime="17.520158326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:24.989319788 +0000 UTC m=+21.751088967" watchObservedRunningTime="2026-09-10 04:41:24.989756002 +0000 UTC m=+21.751525184"
Sep 10 04:41:25.029790 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:25.029736    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-c69jd" podStartSLOduration=11.758321928 podStartE2EDuration="22.029716075s" totalImagesPullingTime="10.271394147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:25.029433489 +0000 UTC m=+21.791202672" watchObservedRunningTime="2026-09-10 04:41:25.029716075 +0000 UTC m=+21.791485257"
Sep 10 04:41:25.067165 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:25.067115    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8r87c" podStartSLOduration=4.448665905 podStartE2EDuration="22.06710039s" totalImagesPullingTime="17.618434485s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:25.067022458 +0000 UTC m=+21.828791642" watchObservedRunningTime="2026-09-10 04:41:25.06710039 +0000 UTC m=+21.828869569"
Sep 10 04:41:25.067586 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:25.067566    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-psjfv" podStartSLOduration=3.562263587 podStartE2EDuration="21.067558715s" totalImagesPullingTime="17.505295128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:25.047056924 +0000 UTC m=+21.808826104" watchObservedRunningTime="2026-09-10 04:41:25.067558715 +0000 UTC m=+21.829327896"
Sep 10 04:41:25.861436 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:25.861413    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:25.861436 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:25.861424    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:25.861638 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:25.861444    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:25.861638 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:25.861531    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclmn" podUID="920551bb-5e01-45e3-80cd-87f46761fb29"
Sep 10 04:41:25.861748 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:25.861652    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47tk5" podUID="7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8"
Sep 10 04:41:25.861803 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:25.861747    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7krfk" podUID="4bbb6665-dfae-48eb-9f23-f5147f2ccebb"
Sep 10 04:41:25.873262 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:25.873237    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 04:41:25.980851 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:25.980731    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr" event={"ID":"a75e8f96-1b83-4636-9a7d-57a5892e5e2b","Type":"ContainerStarted","Data":"30d8133d5b652877c1b8ee95a56e542e910425912f19df3415b016af31e175af"}
Sep 10 04:41:25.982483 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:25.982453    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wg9t7" event={"ID":"47240839-28f9-4291-b91f-b22c142ce1f0","Type":"ContainerStarted","Data":"d9967db77e0568c8d5eaefcb96d6d624efdbe7277f73263da701bb3f0d3cac5b"}
Sep 10 04:41:26.825441 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:26.825230    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T04:41:25.873256853Z","UUID":"0499be29-d393-4273-97ff-274b80706c56","Handler":null,"Name":"","Endpoint":""}
Sep 10 04:41:26.827180 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:26.827156    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 04:41:26.827291 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:26.827197    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 04:41:26.835142 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:26.835110    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-c69jd"
Sep 10 04:41:26.836110 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:26.836086    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-c69jd"
Sep 10 04:41:26.845190 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:26.845165    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c69jd"
Sep 10 04:41:26.845695 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:26.845673    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c69jd"
Sep 10 04:41:26.852320 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:26.852160    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wg9t7" podStartSLOduration=6.33815061 podStartE2EDuration="23.852133956s" totalImagesPullingTime="17.513983346s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:25.996845738 +0000 UTC m=+22.758614922" watchObservedRunningTime="2026-09-10 04:41:26.852133956 +0000 UTC m=+23.613903134"
Sep 10 04:41:27.860208 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:27.860174    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:27.860707 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:27.860286    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:27.860707 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:27.860302    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclmn" podUID="920551bb-5e01-45e3-80cd-87f46761fb29"
Sep 10 04:41:27.860707 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:27.860325    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:27.860707 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:27.860412    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47tk5" podUID="7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8"
Sep 10 04:41:27.860707 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:27.860492    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7krfk" podUID="4bbb6665-dfae-48eb-9f23-f5147f2ccebb"
Sep 10 04:41:27.988936 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:27.988902    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr" event={"ID":"a75e8f96-1b83-4636-9a7d-57a5892e5e2b","Type":"ContainerStarted","Data":"4ac1989720285f3000c7c935e52d9dd0a04f4abee0c24ef3bffa1b4c3ddb9f40"}
Sep 10 04:41:27.992549 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:27.992518    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2hd2_9d909143-4c70-4fe1-900a-3cd520db32e5/ovn-acl-logging/0.log"
Sep 10 04:41:27.992996 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:27.992969    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2" event={"ID":"9d909143-4c70-4fe1-900a-3cd520db32e5","Type":"ContainerStarted","Data":"e91325809d88fde2a82e621bd8351ed6ed9e835344dd0ba1749176b3e566e92e"}
Sep 10 04:41:28.005753 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:28.005710    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b78sr" podStartSLOduration=4.120745115 podStartE2EDuration="24.005696486s" totalImagesPullingTime="19.884951371s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:28.005667353 +0000 UTC m=+24.767436539" watchObservedRunningTime="2026-09-10 04:41:28.005696486 +0000 UTC m=+24.767465649"
Sep 10 04:41:29.857442 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:29.857403    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:29.857442 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:29.857433    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:29.857939 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:29.857403    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:29.857939 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:29.857537    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclmn" podUID="920551bb-5e01-45e3-80cd-87f46761fb29"
Sep 10 04:41:29.857939 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:29.857617    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7krfk" podUID="4bbb6665-dfae-48eb-9f23-f5147f2ccebb"
Sep 10 04:41:29.857939 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:29.857706    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47tk5" podUID="7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8"
Sep 10 04:41:30.000889 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:30.000729    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2hd2_9d909143-4c70-4fe1-900a-3cd520db32e5/ovn-acl-logging/0.log"
Sep 10 04:41:30.001312 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:30.001287    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2" event={"ID":"9d909143-4c70-4fe1-900a-3cd520db32e5","Type":"ContainerStarted","Data":"135485827c407aeb2e20e06022a8eda1f7b5ba73bba793a51b18dec99c879984"}
Sep 10 04:41:31.005566 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:31.005529    2575 generic.go:356] "Generic (PLEG): container finished" podID="dafe7236-9f57-4f88-97d5-b30b2f8ee269" containerID="68ef7a1dda87477977652523a0e5348d572630aa18a3fb9d3fe01fb238ffcf52" exitCode=0
Sep 10 04:41:31.006298 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:31.005596    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ms5c5" event={"ID":"dafe7236-9f57-4f88-97d5-b30b2f8ee269","Type":"ContainerDied","Data":"68ef7a1dda87477977652523a0e5348d572630aa18a3fb9d3fe01fb238ffcf52"}
Sep 10 04:41:31.006298 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:31.005837    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:31.006298 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:31.005897    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:31.006298 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:31.005936    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:31.006298 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:31.006015    2575 scope.go:117] "RemoveContainer" containerID="c93b62ddb98fef9dc1bc673c647e2be107e519d9e552ee26a6864acfe98302c0"
Sep 10 04:41:31.021638 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:31.021614    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:31.021812 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:31.021799    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:41:31.857161 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:31.856932    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:31.857356 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:31.856933    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:31.857356 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:31.856933    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:31.857356 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:31.857282    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7krfk" podUID="4bbb6665-dfae-48eb-9f23-f5147f2ccebb"
Sep 10 04:41:31.857356 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:31.857334    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47tk5" podUID="7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8"
Sep 10 04:41:31.857538 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:31.857421    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclmn" podUID="920551bb-5e01-45e3-80cd-87f46761fb29"
Sep 10 04:41:31.883767 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:31.883738    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pclmn"]
Sep 10 04:41:31.887340 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:31.887311    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-47tk5"]
Sep 10 04:41:31.888085 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:31.888042    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7krfk"]
Sep 10 04:41:32.009526 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:32.009497    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ms5c5" event={"ID":"dafe7236-9f57-4f88-97d5-b30b2f8ee269","Type":"ContainerStarted","Data":"ef2d87f40292168ef2d6f517c634b720a50420f3bd5e0dd6cb9fe19028b46fb3"}
Sep 10 04:41:32.017073 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:32.017042    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2hd2_9d909143-4c70-4fe1-900a-3cd520db32e5/ovn-acl-logging/0.log"
Sep 10 04:41:32.017553 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:32.017534    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:32.017643 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:32.017553    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2" event={"ID":"9d909143-4c70-4fe1-900a-3cd520db32e5","Type":"ContainerStarted","Data":"30243291e3518d125859338d1ead983bb63f665fe03d5effacc5b6927390863a"}
Sep 10 04:41:32.017643 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:32.017573    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:32.017643 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:32.017640    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:32.017973 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:32.017650    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7krfk" podUID="4bbb6665-dfae-48eb-9f23-f5147f2ccebb"
Sep 10 04:41:32.017973 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:32.017873    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47tk5" podUID="7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8"
Sep 10 04:41:32.018131 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:32.018000    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclmn" podUID="920551bb-5e01-45e3-80cd-87f46761fb29"
Sep 10 04:41:32.056561 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:32.056513    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2" podStartSLOduration=11.519402432 podStartE2EDuration="29.05649706s" totalImagesPullingTime="17.537094628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:32.054973735 +0000 UTC m=+28.816742917" watchObservedRunningTime="2026-09-10 04:41:32.05649706 +0000 UTC m=+28.818266243"
Sep 10 04:41:33.021909 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:33.021876    2575 generic.go:356] "Generic (PLEG): container finished" podID="dafe7236-9f57-4f88-97d5-b30b2f8ee269" containerID="ef2d87f40292168ef2d6f517c634b720a50420f3bd5e0dd6cb9fe19028b46fb3" exitCode=0
Sep 10 04:41:33.022339 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:33.021962    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ms5c5" event={"ID":"dafe7236-9f57-4f88-97d5-b30b2f8ee269","Type":"ContainerDied","Data":"ef2d87f40292168ef2d6f517c634b720a50420f3bd5e0dd6cb9fe19028b46fb3"}
Sep 10 04:41:33.827958 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:33.827918    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:33.857715 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:33.857688    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:33.857852 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:33.857772    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47tk5" podUID="7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8"
Sep 10 04:41:33.857910 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:33.857847    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:33.857954 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:33.857932    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:33.858012 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:33.857967    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7krfk" podUID="4bbb6665-dfae-48eb-9f23-f5147f2ccebb"
Sep 10 04:41:33.858012 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:33.857982    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclmn" podUID="920551bb-5e01-45e3-80cd-87f46761fb29"
Sep 10 04:41:33.907474 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:33.907442    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:35.028914 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:35.028883    2575 generic.go:356] "Generic (PLEG): container finished" podID="dafe7236-9f57-4f88-97d5-b30b2f8ee269" containerID="973ee69eec2a493ec836d5eb03ee95e17ae8c40abf19cda6400974abbcef915f" exitCode=0
Sep 10 04:41:35.029316 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:35.028940    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ms5c5" event={"ID":"dafe7236-9f57-4f88-97d5-b30b2f8ee269","Type":"ContainerDied","Data":"973ee69eec2a493ec836d5eb03ee95e17ae8c40abf19cda6400974abbcef915f"}
Sep 10 04:41:35.439910 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:35.439823    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:35.501604 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:35.501567    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:35.857584 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:35.857548    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:35.857734 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:35.857556    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:35.857734 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:35.857677    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47tk5" podUID="7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8"
Sep 10 04:41:35.857734 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:35.857684    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:35.857905 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:35.857796    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7krfk" podUID="4bbb6665-dfae-48eb-9f23-f5147f2ccebb"
Sep 10 04:41:35.857958 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:35.857927    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclmn" podUID="920551bb-5e01-45e3-80cd-87f46761fb29"
Sep 10 04:41:37.025101 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.025061    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-70.ec2.internal" event="NodeReady"
Sep 10 04:41:37.025520 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.025221    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 04:41:37.085009 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.084840    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mmwjm"]
Sep 10 04:41:37.087860 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.087839    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5694dd86c5-7rqjv"]
Sep 10 04:41:37.088017 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.087984    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mmwjm"
Sep 10 04:41:37.090004 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.089985    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h96km"]
Sep 10 04:41:37.090155 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.090127    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.090729 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.090711    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 04:41:37.090837 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.090715    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 04:41:37.090995 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.090977    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nj7q4\""
Sep 10 04:41:37.092469 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.092449    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 04:41:37.092586 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.092517    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 04:41:37.092586 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.092532    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h96km"
Sep 10 04:41:37.092986 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.092966    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 04:41:37.093481 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.093463    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6qv7t\""
Sep 10 04:41:37.098351 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.098003    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 04:41:37.098351 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.098232    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-d4254\""
Sep 10 04:41:37.100265 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.099244    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 04:41:37.100265 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.099556    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mmwjm"]
Sep 10 04:41:37.100265 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.099674    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 04:41:37.100512 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.100497    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h96km"]
Sep 10 04:41:37.101164 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.101143    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 04:41:37.120005 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.119981    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5694dd86c5-7rqjv"]
Sep 10 04:41:37.210709 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.210674    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-bound-sa-token\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.210709 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.210719    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d5779c5-a38f-4b51-841f-896b6c7d873b-cert\") pod \"ingress-canary-h96km\" (UID: \"3d5779c5-a38f-4b51-841f-896b6c7d873b\") " pod="openshift-ingress-canary/ingress-canary-h96km"
Sep 10 04:41:37.210964 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.210753    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b748ea1b-814f-474d-9991-2668883967a3-metrics-tls\") pod \"dns-default-mmwjm\" (UID: \"b748ea1b-814f-474d-9991-2668883967a3\") " pod="openshift-dns/dns-default-mmwjm"
Sep 10 04:41:37.210964 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.210795    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/340229b7-36e5-4ce0-8220-86cf754194e0-installation-pull-secrets\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.210964 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.210825    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmsmk\" (UniqueName: \"kubernetes.io/projected/3d5779c5-a38f-4b51-841f-896b6c7d873b-kube-api-access-bmsmk\") pod \"ingress-canary-h96km\" (UID: \"3d5779c5-a38f-4b51-841f-896b6c7d873b\") " pod="openshift-ingress-canary/ingress-canary-h96km"
Sep 10 04:41:37.210964 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.210851    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/340229b7-36e5-4ce0-8220-86cf754194e0-registry-certificates\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.210964 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.210871    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b748ea1b-814f-474d-9991-2668883967a3-tmp-dir\") pod \"dns-default-mmwjm\" (UID: \"b748ea1b-814f-474d-9991-2668883967a3\") " pod="openshift-dns/dns-default-mmwjm"
Sep 10 04:41:37.210964 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.210942    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/340229b7-36e5-4ce0-8220-86cf754194e0-ca-trust-extracted\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.211288 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.211012    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/340229b7-36e5-4ce0-8220-86cf754194e0-trusted-ca\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.211288 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.211045    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdrs4\" (UniqueName: \"kubernetes.io/projected/b748ea1b-814f-474d-9991-2668883967a3-kube-api-access-cdrs4\") pod \"dns-default-mmwjm\" (UID: \"b748ea1b-814f-474d-9991-2668883967a3\") " pod="openshift-dns/dns-default-mmwjm"
Sep 10 04:41:37.211288 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.211084    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptnlm\" (UniqueName: \"kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-kube-api-access-ptnlm\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.211288 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.211105    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b748ea1b-814f-474d-9991-2668883967a3-config-volume\") pod \"dns-default-mmwjm\" (UID: \"b748ea1b-814f-474d-9991-2668883967a3\") " pod="openshift-dns/dns-default-mmwjm"
Sep 10 04:41:37.211288 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.211132    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-registry-tls\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.211288 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.211200    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/340229b7-36e5-4ce0-8220-86cf754194e0-image-registry-private-configuration\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.211967 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.211940    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:37.270492 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.270456    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:37.312400 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.312369    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-registry-tls\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.312569 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.312417    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/340229b7-36e5-4ce0-8220-86cf754194e0-image-registry-private-configuration\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.312569 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.312457    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-bound-sa-token\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.312569 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.312483    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d5779c5-a38f-4b51-841f-896b6c7d873b-cert\") pod \"ingress-canary-h96km\" (UID: \"3d5779c5-a38f-4b51-841f-896b6c7d873b\") " pod="openshift-ingress-canary/ingress-canary-h96km"
Sep 10 04:41:37.312569 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.312515    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b748ea1b-814f-474d-9991-2668883967a3-metrics-tls\") pod \"dns-default-mmwjm\" (UID: \"b748ea1b-814f-474d-9991-2668883967a3\") " pod="openshift-dns/dns-default-mmwjm"
Sep 10 04:41:37.312569 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:37.312540    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 04:41:37.312569 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:37.312564    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5694dd86c5-7rqjv: secret "image-registry-tls" not found
Sep 10 04:41:37.312844 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:37.312633    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-registry-tls podName:340229b7-36e5-4ce0-8220-86cf754194e0 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:37.812609129 +0000 UTC m=+34.574378306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-registry-tls") pod "image-registry-5694dd86c5-7rqjv" (UID: "340229b7-36e5-4ce0-8220-86cf754194e0") : secret "image-registry-tls" not found
Sep 10 04:41:37.312899 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.312546    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/340229b7-36e5-4ce0-8220-86cf754194e0-installation-pull-secrets\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.312949 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.312923    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmsmk\" (UniqueName: \"kubernetes.io/projected/3d5779c5-a38f-4b51-841f-896b6c7d873b-kube-api-access-bmsmk\") pod \"ingress-canary-h96km\" (UID: \"3d5779c5-a38f-4b51-841f-896b6c7d873b\") " pod="openshift-ingress-canary/ingress-canary-h96km"
Sep 10 04:41:37.313344 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:37.313164    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 04:41:37.313344 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:37.313216    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b748ea1b-814f-474d-9991-2668883967a3-metrics-tls podName:b748ea1b-814f-474d-9991-2668883967a3 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:37.813199316 +0000 UTC m=+34.574968481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b748ea1b-814f-474d-9991-2668883967a3-metrics-tls") pod "dns-default-mmwjm" (UID: "b748ea1b-814f-474d-9991-2668883967a3") : secret "dns-default-metrics-tls" not found
Sep 10 04:41:37.313344 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.313253    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/340229b7-36e5-4ce0-8220-86cf754194e0-registry-certificates\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.313344 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.313291    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b748ea1b-814f-474d-9991-2668883967a3-tmp-dir\") pod \"dns-default-mmwjm\" (UID: \"b748ea1b-814f-474d-9991-2668883967a3\") " pod="openshift-dns/dns-default-mmwjm"
Sep 10 04:41:37.313344 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.313335    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/340229b7-36e5-4ce0-8220-86cf754194e0-ca-trust-extracted\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.313648 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.313378    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/340229b7-36e5-4ce0-8220-86cf754194e0-trusted-ca\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.313648 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.313406    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdrs4\" (UniqueName: \"kubernetes.io/projected/b748ea1b-814f-474d-9991-2668883967a3-kube-api-access-cdrs4\") pod \"dns-default-mmwjm\" (UID: \"b748ea1b-814f-474d-9991-2668883967a3\") " pod="openshift-dns/dns-default-mmwjm"
Sep 10 04:41:37.313648 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.313424    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptnlm\" (UniqueName: \"kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-kube-api-access-ptnlm\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.313648 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.313442    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b748ea1b-814f-474d-9991-2668883967a3-config-volume\") pod \"dns-default-mmwjm\" (UID: \"b748ea1b-814f-474d-9991-2668883967a3\") " pod="openshift-dns/dns-default-mmwjm"
Sep 10 04:41:37.313839 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.313655    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b748ea1b-814f-474d-9991-2668883967a3-tmp-dir\") pod \"dns-default-mmwjm\" (UID: \"b748ea1b-814f-474d-9991-2668883967a3\") " pod="openshift-dns/dns-default-mmwjm"
Sep 10 04:41:37.314107 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.313987    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/340229b7-36e5-4ce0-8220-86cf754194e0-registry-certificates\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.314216 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.314192    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b748ea1b-814f-474d-9991-2668883967a3-config-volume\") pod \"dns-default-mmwjm\" (UID: \"b748ea1b-814f-474d-9991-2668883967a3\") " pod="openshift-dns/dns-default-mmwjm"
Sep 10 04:41:37.314357 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.314336    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/340229b7-36e5-4ce0-8220-86cf754194e0-ca-trust-extracted\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.314511 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.314484    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/340229b7-36e5-4ce0-8220-86cf754194e0-trusted-ca\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.317114 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.316965    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/340229b7-36e5-4ce0-8220-86cf754194e0-image-registry-private-configuration\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.317199 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.316980    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/340229b7-36e5-4ce0-8220-86cf754194e0-installation-pull-secrets\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.317199 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.317027    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d5779c5-a38f-4b51-841f-896b6c7d873b-cert\") pod \"ingress-canary-h96km\" (UID: \"3d5779c5-a38f-4b51-841f-896b6c7d873b\") " pod="openshift-ingress-canary/ingress-canary-h96km"
Sep 10 04:41:37.324789 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.324759    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdrs4\" (UniqueName: \"kubernetes.io/projected/b748ea1b-814f-474d-9991-2668883967a3-kube-api-access-cdrs4\") pod \"dns-default-mmwjm\" (UID: \"b748ea1b-814f-474d-9991-2668883967a3\") " pod="openshift-dns/dns-default-mmwjm"
Sep 10 04:41:37.324968 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.324945    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptnlm\" (UniqueName: \"kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-kube-api-access-ptnlm\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.325187 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.325169    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-bound-sa-token\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.325293 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.325276    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmsmk\" (UniqueName: \"kubernetes.io/projected/3d5779c5-a38f-4b51-841f-896b6c7d873b-kube-api-access-bmsmk\") pod \"ingress-canary-h96km\" (UID: \"3d5779c5-a38f-4b51-841f-896b6c7d873b\") " pod="openshift-ingress-canary/ingress-canary-h96km"
Sep 10 04:41:37.414732 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.414651    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret\") pod \"global-pull-secret-syncer-pclmn\" (UID: \"920551bb-5e01-45e3-80cd-87f46761fb29\") " pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:37.414732 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.414697    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs\") pod \"network-metrics-daemon-7krfk\" (UID: \"4bbb6665-dfae-48eb-9f23-f5147f2ccebb\") " pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:37.414914 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:37.414802    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 04:41:37.414914 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:37.414865    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs podName:4bbb6665-dfae-48eb-9f23-f5147f2ccebb nodeName:}" failed. No retries permitted until 2026-09-10 04:42:09.414851265 +0000 UTC m=+66.176620425 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs") pod "network-metrics-daemon-7krfk" (UID: "4bbb6665-dfae-48eb-9f23-f5147f2ccebb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 04:41:37.414914 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:37.414803    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 04:41:37.415023 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:37.414918    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret podName:920551bb-5e01-45e3-80cd-87f46761fb29 nodeName:}" failed. No retries permitted until 2026-09-10 04:42:09.414906635 +0000 UTC m=+66.176675803 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret") pod "global-pull-secret-syncer-pclmn" (UID: "920551bb-5e01-45e3-80cd-87f46761fb29") : object "kube-system"/"original-pull-secret" not registered
Sep 10 04:41:37.416592 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.416574    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h96km"
Sep 10 04:41:37.515584 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.515547    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7hpx\" (UniqueName: \"kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx\") pod \"network-check-target-47tk5\" (UID: \"7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8\") " pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:37.515735 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:37.515706    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 04:41:37.515735 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:37.515725    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 04:41:37.515735 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:37.515736    2575 projected.go:196] Error preparing data for projected volume kube-api-access-r7hpx for pod openshift-network-diagnostics/network-check-target-47tk5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 04:41:37.515873 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:37.515787    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx podName:7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8 nodeName:}" failed. No retries permitted until 2026-09-10 04:42:09.51577279 +0000 UTC m=+66.277541969 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-r7hpx" (UniqueName: "kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx") pod "network-check-target-47tk5" (UID: "7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 04:41:37.694300 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.694225    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h96km"]
Sep 10 04:41:37.818791 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.818758    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-registry-tls\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:37.818960 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.818807    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b748ea1b-814f-474d-9991-2668883967a3-metrics-tls\") pod \"dns-default-mmwjm\" (UID: \"b748ea1b-814f-474d-9991-2668883967a3\") " pod="openshift-dns/dns-default-mmwjm"
Sep 10 04:41:37.818960 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:37.818929    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 04:41:37.818960 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:37.818936    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 04:41:37.818960 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:37.818961    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5694dd86c5-7rqjv: secret "image-registry-tls" not found
Sep 10 04:41:37.819193 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:37.818994    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b748ea1b-814f-474d-9991-2668883967a3-metrics-tls podName:b748ea1b-814f-474d-9991-2668883967a3 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:38.818974681 +0000 UTC m=+35.580743848 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b748ea1b-814f-474d-9991-2668883967a3-metrics-tls") pod "dns-default-mmwjm" (UID: "b748ea1b-814f-474d-9991-2668883967a3") : secret "dns-default-metrics-tls" not found
Sep 10 04:41:37.819193 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:37.819012    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-registry-tls podName:340229b7-36e5-4ce0-8220-86cf754194e0 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:38.81900311 +0000 UTC m=+35.580772269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-registry-tls") pod "image-registry-5694dd86c5-7rqjv" (UID: "340229b7-36e5-4ce0-8220-86cf754194e0") : secret "image-registry-tls" not found
Sep 10 04:41:37.857297 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.857263    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:41:37.857458 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.857263    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:41:37.857458 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.857416    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:41:37.860089 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.860000    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 04:41:37.860377 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.860357    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 04:41:37.860517 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.860439    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jksxr\""
Sep 10 04:41:37.860517 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.860491    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 04:41:37.860517 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.860440    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 04:41:37.860703 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:37.860540    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gx84n\""
Sep 10 04:41:38.829549 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:38.829510    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b748ea1b-814f-474d-9991-2668883967a3-metrics-tls\") pod \"dns-default-mmwjm\" (UID: \"b748ea1b-814f-474d-9991-2668883967a3\") " pod="openshift-dns/dns-default-mmwjm"
Sep 10 04:41:38.830225 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:38.829616    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-registry-tls\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:38.830225 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:38.829693    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 04:41:38.830225 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:38.829718    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 04:41:38.830225 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:38.829729    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5694dd86c5-7rqjv: secret "image-registry-tls" not found
Sep 10 04:41:38.830225 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:38.829777    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b748ea1b-814f-474d-9991-2668883967a3-metrics-tls podName:b748ea1b-814f-474d-9991-2668883967a3 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:40.82975631 +0000 UTC m=+37.591525545 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b748ea1b-814f-474d-9991-2668883967a3-metrics-tls") pod "dns-default-mmwjm" (UID: "b748ea1b-814f-474d-9991-2668883967a3") : secret "dns-default-metrics-tls" not found
Sep 10 04:41:38.830225 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:38.829791    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-registry-tls podName:340229b7-36e5-4ce0-8220-86cf754194e0 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:40.829785597 +0000 UTC m=+37.591554756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-registry-tls") pod "image-registry-5694dd86c5-7rqjv" (UID: "340229b7-36e5-4ce0-8220-86cf754194e0") : secret "image-registry-tls" not found
Sep 10 04:41:38.975001 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:38.974972    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5779c5_a38f_4b51_841f_896b6c7d873b.slice/crio-aa325f62b8687109d37234d1ac9bfbb9754f7d0db3f8fe426eb9fa33105f2fb0 WatchSource:0}: Error finding container aa325f62b8687109d37234d1ac9bfbb9754f7d0db3f8fe426eb9fa33105f2fb0: Status 404 returned error can't find the container with id aa325f62b8687109d37234d1ac9bfbb9754f7d0db3f8fe426eb9fa33105f2fb0
Sep 10 04:41:39.039119 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:39.039091    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h96km" event={"ID":"3d5779c5-a38f-4b51-841f-896b6c7d873b","Type":"ContainerStarted","Data":"aa325f62b8687109d37234d1ac9bfbb9754f7d0db3f8fe426eb9fa33105f2fb0"}
Sep 10 04:41:39.142937 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:39.142905    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:39.250218 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:39.250166    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:40.045203 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:40.044848    2575 generic.go:356] "Generic (PLEG): container finished" podID="dafe7236-9f57-4f88-97d5-b30b2f8ee269" containerID="6c7bdd0d1621fd0ab585661b8a7d19798ee7496bc06d306c48a8576ce1262c37" exitCode=0
Sep 10 04:41:40.045203 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:40.044918    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ms5c5" event={"ID":"dafe7236-9f57-4f88-97d5-b30b2f8ee269","Type":"ContainerDied","Data":"6c7bdd0d1621fd0ab585661b8a7d19798ee7496bc06d306c48a8576ce1262c37"}
Sep 10 04:41:40.848557 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:40.848520    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b748ea1b-814f-474d-9991-2668883967a3-metrics-tls\") pod \"dns-default-mmwjm\" (UID: \"b748ea1b-814f-474d-9991-2668883967a3\") " pod="openshift-dns/dns-default-mmwjm"
Sep 10 04:41:40.848722 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:40.848612    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-registry-tls\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:40.848722 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:40.848667    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 04:41:40.848722 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:40.848712    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 04:41:40.848866 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:40.848728    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5694dd86c5-7rqjv: secret "image-registry-tls" not found
Sep 10 04:41:40.848866 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:40.848737    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b748ea1b-814f-474d-9991-2668883967a3-metrics-tls podName:b748ea1b-814f-474d-9991-2668883967a3 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:44.848722188 +0000 UTC m=+41.610491369 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b748ea1b-814f-474d-9991-2668883967a3-metrics-tls") pod "dns-default-mmwjm" (UID: "b748ea1b-814f-474d-9991-2668883967a3") : secret "dns-default-metrics-tls" not found
Sep 10 04:41:40.848866 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:40.848774    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-registry-tls podName:340229b7-36e5-4ce0-8220-86cf754194e0 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:44.848759987 +0000 UTC m=+41.610529147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-registry-tls") pod "image-registry-5694dd86c5-7rqjv" (UID: "340229b7-36e5-4ce0-8220-86cf754194e0") : secret "image-registry-tls" not found
Sep 10 04:41:41.051673 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:41.051636    2575 generic.go:356] "Generic (PLEG): container finished" podID="dafe7236-9f57-4f88-97d5-b30b2f8ee269" containerID="2a4b86f732a7965de5d6d773cb4a3493ba8806b1097c45ed09548045d2d605d2" exitCode=0
Sep 10 04:41:41.052134 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:41.051725    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ms5c5" event={"ID":"dafe7236-9f57-4f88-97d5-b30b2f8ee269","Type":"ContainerDied","Data":"2a4b86f732a7965de5d6d773cb4a3493ba8806b1097c45ed09548045d2d605d2"}
Sep 10 04:41:41.807623 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:41.807583    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:41.908713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:41.908633    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:42.060229 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:42.060191    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ms5c5" event={"ID":"dafe7236-9f57-4f88-97d5-b30b2f8ee269","Type":"ContainerStarted","Data":"d1e4a4a1d249d12223c032b854e85cf0e83551896742d40d358731f0364ab4dc"}
Sep 10 04:41:42.061579 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:42.061555    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h96km" event={"ID":"3d5779c5-a38f-4b51-841f-896b6c7d873b","Type":"ContainerStarted","Data":"37029f97ad7cda9501924f9b43533d8e8c09261265fbb7885a30ac6a719ab22b"}
Sep 10 04:41:42.087322 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:42.087279    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ms5c5" podStartSLOduration=10.293693125 podStartE2EDuration="39.08726531s" totalImagesPullingTime="28.793572185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:42.085751665 +0000 UTC m=+38.847520847" watchObservedRunningTime="2026-09-10 04:41:42.08726531 +0000 UTC m=+38.849034491"
Sep 10 04:41:42.101121 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:42.101052    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h96km" podStartSLOduration=2.486755991 podStartE2EDuration="5.101037501s" totalImagesPullingTime="2.61428151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:42.100509567 +0000 UTC m=+38.862278749" watchObservedRunningTime="2026-09-10 04:41:42.101037501 +0000 UTC m=+38.862806708"
Sep 10 04:41:42.200652 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:42.200620    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:42.270717 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:42.270685    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:42.369398 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:42.369362    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:42.439415 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:42.439381    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:43.296704 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:43.296672    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:44.209307 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:44.209273    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:44.364448 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:44.364406    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:44.504192 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:44.504085    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:44.599164 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:44.599133    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:44.708159 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:44.708131    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:44.724240 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:44.724216    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nwwqh"]
Sep 10 04:41:44.763534 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:44.763467    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nwwqh"]
Sep 10 04:41:44.763643 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:44.763566    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nwwqh"
Sep 10 04:41:44.769446 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:44.769413    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 10 04:41:44.769564 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:44.769415    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 10 04:41:44.769564 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:44.769535    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-q5rbn\""
Sep 10 04:41:44.822964 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:44.822926    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:44.879320 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:44.879292    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-registry-tls\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:44.879440 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:44.879344    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b748ea1b-814f-474d-9991-2668883967a3-metrics-tls\") pod \"dns-default-mmwjm\" (UID: \"b748ea1b-814f-474d-9991-2668883967a3\") " pod="openshift-dns/dns-default-mmwjm"
Sep 10 04:41:44.879440 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:44.879375    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b565t\" (UniqueName: \"kubernetes.io/projected/c16defdf-d7ad-44ac-abfe-8c3d4f2199f9-kube-api-access-b565t\") pod \"migrator-fdbb8fdff-nwwqh\" (UID: \"c16defdf-d7ad-44ac-abfe-8c3d4f2199f9\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nwwqh"
Sep 10 04:41:44.879507 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:44.879445    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 04:41:44.879507 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:44.879461    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5694dd86c5-7rqjv: secret "image-registry-tls" not found
Sep 10 04:41:44.879507 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:44.879469    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 04:41:44.879602 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:44.879515    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-registry-tls podName:340229b7-36e5-4ce0-8220-86cf754194e0 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:52.87950088 +0000 UTC m=+49.641270045 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-registry-tls") pod "image-registry-5694dd86c5-7rqjv" (UID: "340229b7-36e5-4ce0-8220-86cf754194e0") : secret "image-registry-tls" not found
Sep 10 04:41:44.879602 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:44.879578    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b748ea1b-814f-474d-9991-2668883967a3-metrics-tls podName:b748ea1b-814f-474d-9991-2668883967a3 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:52.879566923 +0000 UTC m=+49.641336096 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b748ea1b-814f-474d-9991-2668883967a3-metrics-tls") pod "dns-default-mmwjm" (UID: "b748ea1b-814f-474d-9991-2668883967a3") : secret "dns-default-metrics-tls" not found
Sep 10 04:41:44.934340 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:44.934311    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:44.980332 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:44.980301    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b565t\" (UniqueName: \"kubernetes.io/projected/c16defdf-d7ad-44ac-abfe-8c3d4f2199f9-kube-api-access-b565t\") pod \"migrator-fdbb8fdff-nwwqh\" (UID: \"c16defdf-d7ad-44ac-abfe-8c3d4f2199f9\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nwwqh"
Sep 10 04:41:44.988762 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:44.988744    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-srmv2"]
Sep 10 04:41:44.996291 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:44.996266    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b565t\" (UniqueName: \"kubernetes.io/projected/c16defdf-d7ad-44ac-abfe-8c3d4f2199f9-kube-api-access-b565t\") pod \"migrator-fdbb8fdff-nwwqh\" (UID: \"c16defdf-d7ad-44ac-abfe-8c3d4f2199f9\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nwwqh"
Sep 10 04:41:45.009528 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.009506    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-srmv2"]
Sep 10 04:41:45.009618 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.009527    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:45.012140 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.012119    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 04:41:45.012324 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.012138    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 04:41:45.012324 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.012143    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 04:41:45.012523 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.012179    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 04:41:45.012523 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.012197    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kf5vp\""
Sep 10 04:41:45.040518 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.040484    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:45.072840 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.072811    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nwwqh"
Sep 10 04:41:45.182582 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.182548    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6440e031-173d-40dd-8186-b80bb9fb3fdb-data-volume\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:45.182582 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.182580    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6440e031-173d-40dd-8186-b80bb9fb3fdb-crio-socket\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:45.182757 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.182609    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2zqk\" (UniqueName: \"kubernetes.io/projected/6440e031-173d-40dd-8186-b80bb9fb3fdb-kube-api-access-t2zqk\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:45.182757 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.182661    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6440e031-173d-40dd-8186-b80bb9fb3fdb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:45.182757 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.182691    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6440e031-173d-40dd-8186-b80bb9fb3fdb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:45.225839 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.225801    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:45.283763 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.283698    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6440e031-173d-40dd-8186-b80bb9fb3fdb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:45.283763 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.283728    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6440e031-173d-40dd-8186-b80bb9fb3fdb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:45.283903 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.283810    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6440e031-173d-40dd-8186-b80bb9fb3fdb-data-volume\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:45.283903 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.283837    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6440e031-173d-40dd-8186-b80bb9fb3fdb-crio-socket\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:45.283903 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.283875    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2zqk\" (UniqueName: \"kubernetes.io/projected/6440e031-173d-40dd-8186-b80bb9fb3fdb-kube-api-access-t2zqk\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:45.283903 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:45.283812    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 04:41:45.284100 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:45.283963    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6440e031-173d-40dd-8186-b80bb9fb3fdb-kube-rbac-proxy-cm podName:6440e031-173d-40dd-8186-b80bb9fb3fdb nodeName:}" failed. No retries permitted until 2026-09-10 04:41:45.783940712 +0000 UTC m=+42.545709872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6440e031-173d-40dd-8186-b80bb9fb3fdb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-srmv2" (UID: "6440e031-173d-40dd-8186-b80bb9fb3fdb") : configmap "kube-rbac-proxy" not found
Sep 10 04:41:45.284100 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:45.283884    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 04:41:45.284100 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:45.284041    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6440e031-173d-40dd-8186-b80bb9fb3fdb-insights-runtime-extractor-tls podName:6440e031-173d-40dd-8186-b80bb9fb3fdb nodeName:}" failed. No retries permitted until 2026-09-10 04:41:45.784019843 +0000 UTC m=+42.545789017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6440e031-173d-40dd-8186-b80bb9fb3fdb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-srmv2" (UID: "6440e031-173d-40dd-8186-b80bb9fb3fdb") : secret "insights-runtime-extractor-tls" not found
Sep 10 04:41:45.284100 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.284057    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6440e031-173d-40dd-8186-b80bb9fb3fdb-crio-socket\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:45.284257 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.284181    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6440e031-173d-40dd-8186-b80bb9fb3fdb-data-volume\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:45.292857 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.292833    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2zqk\" (UniqueName: \"kubernetes.io/projected/6440e031-173d-40dd-8186-b80bb9fb3fdb-kube-api-access-t2zqk\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:45.296899 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.296874    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:45.362590 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.362564    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nwwqh"]
Sep 10 04:41:45.365440 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:45.365417    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16defdf_d7ad_44ac_abfe_8c3d4f2199f9.slice/crio-30815d837cf1e8da70d613af7743e9f46c8faf630104928a9a468b4c96a8794c WatchSource:0}: Error finding container 30815d837cf1e8da70d613af7743e9f46c8faf630104928a9a468b4c96a8794c: Status 404 returned error can't find the container with id 30815d837cf1e8da70d613af7743e9f46c8faf630104928a9a468b4c96a8794c
Sep 10 04:41:45.788328 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.788297    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6440e031-173d-40dd-8186-b80bb9fb3fdb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:45.788494 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:45.788333    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6440e031-173d-40dd-8186-b80bb9fb3fdb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:45.788494 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:45.788415    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 04:41:45.788494 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:45.788463    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 04:41:45.788494 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:45.788484    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6440e031-173d-40dd-8186-b80bb9fb3fdb-kube-rbac-proxy-cm podName:6440e031-173d-40dd-8186-b80bb9fb3fdb nodeName:}" failed. No retries permitted until 2026-09-10 04:41:46.788466698 +0000 UTC m=+43.550235858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6440e031-173d-40dd-8186-b80bb9fb3fdb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-srmv2" (UID: "6440e031-173d-40dd-8186-b80bb9fb3fdb") : configmap "kube-rbac-proxy" not found
Sep 10 04:41:45.788626 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:45.788509    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6440e031-173d-40dd-8186-b80bb9fb3fdb-insights-runtime-extractor-tls podName:6440e031-173d-40dd-8186-b80bb9fb3fdb nodeName:}" failed. No retries permitted until 2026-09-10 04:41:46.78849815 +0000 UTC m=+43.550267328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6440e031-173d-40dd-8186-b80bb9fb3fdb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-srmv2" (UID: "6440e031-173d-40dd-8186-b80bb9fb3fdb") : secret "insights-runtime-extractor-tls" not found
Sep 10 04:41:46.071724 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.071641    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nwwqh" event={"ID":"c16defdf-d7ad-44ac-abfe-8c3d4f2199f9","Type":"ContainerStarted","Data":"30815d837cf1e8da70d613af7743e9f46c8faf630104928a9a468b4c96a8794c"}
Sep 10 04:41:46.295554 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.295515    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:46.380120 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.380023    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:46.455648 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.455612    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:46.503426 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.503399    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-n2rcz"]
Sep 10 04:41:46.520882 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.520858    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-n2rcz"]
Sep 10 04:41:46.520992 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.520947    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-n2rcz"
Sep 10 04:41:46.523476 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.523454    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 10 04:41:46.523580 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.523492    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 10 04:41:46.523580 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.523520    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-4hps8\""
Sep 10 04:41:46.540293 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.540269    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:46.696163 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.696098    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29a6dafa-c02d-4b50-a299-f6b873560b52-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-n2rcz\" (UID: \"29a6dafa-c02d-4b50-a299-f6b873560b52\") " pod="openshift-network-console/networking-console-plugin-79977f9844-n2rcz"
Sep 10 04:41:46.696283 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.696199    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/29a6dafa-c02d-4b50-a299-f6b873560b52-nginx-conf\") pod \"networking-console-plugin-79977f9844-n2rcz\" (UID: \"29a6dafa-c02d-4b50-a299-f6b873560b52\") " pod="openshift-network-console/networking-console-plugin-79977f9844-n2rcz"
Sep 10 04:41:46.737828 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.737796    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-jcl9d"]
Sep 10 04:41:46.760972 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.760944    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-jcl9d"]
Sep 10 04:41:46.761187 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.761097    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-jcl9d"
Sep 10 04:41:46.765694 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.765655    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 10 04:41:46.765793 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.765695    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 10 04:41:46.765986 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.765920    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xj7sz\""
Sep 10 04:41:46.765986 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.765939    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 10 04:41:46.765986 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.765956    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 10 04:41:46.771908 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.771877    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:46.796826 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.796807    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29a6dafa-c02d-4b50-a299-f6b873560b52-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-n2rcz\" (UID: \"29a6dafa-c02d-4b50-a299-f6b873560b52\") " pod="openshift-network-console/networking-console-plugin-79977f9844-n2rcz"
Sep 10 04:41:46.796932 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.796842    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6440e031-173d-40dd-8186-b80bb9fb3fdb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:46.796932 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.796861    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6440e031-173d-40dd-8186-b80bb9fb3fdb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:46.797046 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:46.796951    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 04:41:46.797046 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:46.796968    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 04:41:46.797046 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.797000    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/29a6dafa-c02d-4b50-a299-f6b873560b52-nginx-conf\") pod \"networking-console-plugin-79977f9844-n2rcz\" (UID: \"29a6dafa-c02d-4b50-a299-f6b873560b52\") " pod="openshift-network-console/networking-console-plugin-79977f9844-n2rcz"
Sep 10 04:41:46.797046 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:46.797007    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6440e031-173d-40dd-8186-b80bb9fb3fdb-kube-rbac-proxy-cm podName:6440e031-173d-40dd-8186-b80bb9fb3fdb nodeName:}" failed. No retries permitted until 2026-09-10 04:41:48.796992442 +0000 UTC m=+45.558761608 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6440e031-173d-40dd-8186-b80bb9fb3fdb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-srmv2" (UID: "6440e031-173d-40dd-8186-b80bb9fb3fdb") : configmap "kube-rbac-proxy" not found
Sep 10 04:41:46.797046 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:46.796969    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 04:41:46.797293 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:46.797091    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6440e031-173d-40dd-8186-b80bb9fb3fdb-insights-runtime-extractor-tls podName:6440e031-173d-40dd-8186-b80bb9fb3fdb nodeName:}" failed. No retries permitted until 2026-09-10 04:41:48.797047936 +0000 UTC m=+45.558817098 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6440e031-173d-40dd-8186-b80bb9fb3fdb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-srmv2" (UID: "6440e031-173d-40dd-8186-b80bb9fb3fdb") : secret "insights-runtime-extractor-tls" not found
Sep 10 04:41:46.797293 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:46.797120    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29a6dafa-c02d-4b50-a299-f6b873560b52-networking-console-plugin-cert podName:29a6dafa-c02d-4b50-a299-f6b873560b52 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:47.297107575 +0000 UTC m=+44.058876741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/29a6dafa-c02d-4b50-a299-f6b873560b52-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-n2rcz" (UID: "29a6dafa-c02d-4b50-a299-f6b873560b52") : secret "networking-console-plugin-cert" not found
Sep 10 04:41:46.797629 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.797611    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/29a6dafa-c02d-4b50-a299-f6b873560b52-nginx-conf\") pod \"networking-console-plugin-79977f9844-n2rcz\" (UID: \"29a6dafa-c02d-4b50-a299-f6b873560b52\") " pod="openshift-network-console/networking-console-plugin-79977f9844-n2rcz"
Sep 10 04:41:46.873942 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.873865    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:46.898226 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.898197    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh64g\" (UniqueName: \"kubernetes.io/projected/0d3567c6-a90d-4be0-a5b2-cec666ae1777-kube-api-access-bh64g\") pod \"service-ca-75d99c484-jcl9d\" (UID: \"0d3567c6-a90d-4be0-a5b2-cec666ae1777\") " pod="openshift-service-ca/service-ca-75d99c484-jcl9d"
Sep 10 04:41:46.898406 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.898326    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0d3567c6-a90d-4be0-a5b2-cec666ae1777-signing-key\") pod \"service-ca-75d99c484-jcl9d\" (UID: \"0d3567c6-a90d-4be0-a5b2-cec666ae1777\") " pod="openshift-service-ca/service-ca-75d99c484-jcl9d"
Sep 10 04:41:46.898406 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.898365    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0d3567c6-a90d-4be0-a5b2-cec666ae1777-signing-cabundle\") pod \"service-ca-75d99c484-jcl9d\" (UID: \"0d3567c6-a90d-4be0-a5b2-cec666ae1777\") " pod="openshift-service-ca/service-ca-75d99c484-jcl9d"
Sep 10 04:41:46.998974 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.998944    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh64g\" (UniqueName: \"kubernetes.io/projected/0d3567c6-a90d-4be0-a5b2-cec666ae1777-kube-api-access-bh64g\") pod \"service-ca-75d99c484-jcl9d\" (UID: \"0d3567c6-a90d-4be0-a5b2-cec666ae1777\") " pod="openshift-service-ca/service-ca-75d99c484-jcl9d"
Sep 10 04:41:46.999163 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.999111    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0d3567c6-a90d-4be0-a5b2-cec666ae1777-signing-key\") pod \"service-ca-75d99c484-jcl9d\" (UID: \"0d3567c6-a90d-4be0-a5b2-cec666ae1777\") " pod="openshift-service-ca/service-ca-75d99c484-jcl9d"
Sep 10 04:41:46.999163 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.999148    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0d3567c6-a90d-4be0-a5b2-cec666ae1777-signing-cabundle\") pod \"service-ca-75d99c484-jcl9d\" (UID: \"0d3567c6-a90d-4be0-a5b2-cec666ae1777\") " pod="openshift-service-ca/service-ca-75d99c484-jcl9d"
Sep 10 04:41:46.999680 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:46.999660    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0d3567c6-a90d-4be0-a5b2-cec666ae1777-signing-cabundle\") pod \"service-ca-75d99c484-jcl9d\" (UID: \"0d3567c6-a90d-4be0-a5b2-cec666ae1777\") " pod="openshift-service-ca/service-ca-75d99c484-jcl9d"
Sep 10 04:41:47.001362 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:47.001346    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0d3567c6-a90d-4be0-a5b2-cec666ae1777-signing-key\") pod \"service-ca-75d99c484-jcl9d\" (UID: \"0d3567c6-a90d-4be0-a5b2-cec666ae1777\") " pod="openshift-service-ca/service-ca-75d99c484-jcl9d"
Sep 10 04:41:47.007154 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:47.007135    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh64g\" (UniqueName: \"kubernetes.io/projected/0d3567c6-a90d-4be0-a5b2-cec666ae1777-kube-api-access-bh64g\") pod \"service-ca-75d99c484-jcl9d\" (UID: \"0d3567c6-a90d-4be0-a5b2-cec666ae1777\") " pod="openshift-service-ca/service-ca-75d99c484-jcl9d"
Sep 10 04:41:47.071193 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:47.071168    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-jcl9d"
Sep 10 04:41:47.146700 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:47.146681    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2kswj_1a1db6ec-8b9d-4f0a-8bf0-de72e290cdd1/dns-node-resolver/0.log"
Sep 10 04:41:47.302293 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:47.302270    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29a6dafa-c02d-4b50-a299-f6b873560b52-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-n2rcz\" (UID: \"29a6dafa-c02d-4b50-a299-f6b873560b52\") " pod="openshift-network-console/networking-console-plugin-79977f9844-n2rcz"
Sep 10 04:41:47.302415 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:47.302396    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 04:41:47.302459 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:47.302453    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29a6dafa-c02d-4b50-a299-f6b873560b52-networking-console-plugin-cert podName:29a6dafa-c02d-4b50-a299-f6b873560b52 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:48.302439266 +0000 UTC m=+45.064208426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/29a6dafa-c02d-4b50-a299-f6b873560b52-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-n2rcz" (UID: "29a6dafa-c02d-4b50-a299-f6b873560b52") : secret "networking-console-plugin-cert" not found
Sep 10 04:41:47.352619 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:47.352591    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-jcl9d"]
Sep 10 04:41:47.355102 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:41:47.355060    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3567c6_a90d_4be0_a5b2_cec666ae1777.slice/crio-d9fb81471e4ef3cfa364b81faef5498c5ccce4c09d465f533f1b146e5a58c299 WatchSource:0}: Error finding container d9fb81471e4ef3cfa364b81faef5498c5ccce4c09d465f533f1b146e5a58c299: Status 404 returned error can't find the container with id d9fb81471e4ef3cfa364b81faef5498c5ccce4c09d465f533f1b146e5a58c299
Sep 10 04:41:47.718774 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:47.718703    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:48.079024 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:48.078940    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nwwqh" event={"ID":"c16defdf-d7ad-44ac-abfe-8c3d4f2199f9","Type":"ContainerStarted","Data":"5d98d64032616fd9eb30b6c84daab899c7f40e15ee2af43845cf4a6a6b1a6425"}
Sep 10 04:41:48.079024 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:48.079008    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nwwqh" event={"ID":"c16defdf-d7ad-44ac-abfe-8c3d4f2199f9","Type":"ContainerStarted","Data":"34372412e6061279dc041223f4a526f0326effde26db2455ca0432dfb37b76af"}
Sep 10 04:41:48.080374 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:48.080329    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-jcl9d" event={"ID":"0d3567c6-a90d-4be0-a5b2-cec666ae1777","Type":"ContainerStarted","Data":"d9fb81471e4ef3cfa364b81faef5498c5ccce4c09d465f533f1b146e5a58c299"}
Sep 10 04:41:48.095652 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:48.095599    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nwwqh" podStartSLOduration=2.166042681 podStartE2EDuration="4.09558196s" totalImagesPullingTime="1.929539279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:48.095283535 +0000 UTC m=+44.857052719" watchObservedRunningTime="2026-09-10 04:41:48.09558196 +0000 UTC m=+44.857351142"
Sep 10 04:41:48.311680 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:48.311643    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29a6dafa-c02d-4b50-a299-f6b873560b52-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-n2rcz\" (UID: \"29a6dafa-c02d-4b50-a299-f6b873560b52\") " pod="openshift-network-console/networking-console-plugin-79977f9844-n2rcz"
Sep 10 04:41:48.311832 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:48.311791    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 04:41:48.311877 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:48.311854    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29a6dafa-c02d-4b50-a299-f6b873560b52-networking-console-plugin-cert podName:29a6dafa-c02d-4b50-a299-f6b873560b52 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:50.311839117 +0000 UTC m=+47.073608277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/29a6dafa-c02d-4b50-a299-f6b873560b52-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-n2rcz" (UID: "29a6dafa-c02d-4b50-a299-f6b873560b52") : secret "networking-console-plugin-cert" not found
Sep 10 04:41:48.543176 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:48.543147    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n2zlm_02eb37ae-67ce-4454-b34a-e3666a73f999/node-ca/0.log"
Sep 10 04:41:48.815487 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:48.815404    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6440e031-173d-40dd-8186-b80bb9fb3fdb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:48.815487 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:48.815439    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6440e031-173d-40dd-8186-b80bb9fb3fdb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:48.815902 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:48.815530    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 04:41:48.815902 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:48.815543    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 04:41:48.815902 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:48.815581    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6440e031-173d-40dd-8186-b80bb9fb3fdb-insights-runtime-extractor-tls podName:6440e031-173d-40dd-8186-b80bb9fb3fdb nodeName:}" failed. No retries permitted until 2026-09-10 04:41:52.815567342 +0000 UTC m=+49.577336514 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6440e031-173d-40dd-8186-b80bb9fb3fdb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-srmv2" (UID: "6440e031-173d-40dd-8186-b80bb9fb3fdb") : secret "insights-runtime-extractor-tls" not found
Sep 10 04:41:48.815902 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:48.815608    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6440e031-173d-40dd-8186-b80bb9fb3fdb-kube-rbac-proxy-cm podName:6440e031-173d-40dd-8186-b80bb9fb3fdb nodeName:}" failed. No retries permitted until 2026-09-10 04:41:52.815589586 +0000 UTC m=+49.577358764 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6440e031-173d-40dd-8186-b80bb9fb3fdb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-srmv2" (UID: "6440e031-173d-40dd-8186-b80bb9fb3fdb") : configmap "kube-rbac-proxy" not found
Sep 10 04:41:48.943292 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:48.943252    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h96km_3d5779c5-a38f-4b51-841f-896b6c7d873b/serve-healthcheck-canary/0.log"
Sep 10 04:41:49.343406 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:49.343370    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-nwwqh_c16defdf-d7ad-44ac-abfe-8c3d4f2199f9/migrator/0.log"
Sep 10 04:41:49.542884 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:49.542854    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-nwwqh_c16defdf-d7ad-44ac-abfe-8c3d4f2199f9/graceful-termination/0.log"
Sep 10 04:41:50.086837 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:50.086751    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-jcl9d" event={"ID":"0d3567c6-a90d-4be0-a5b2-cec666ae1777","Type":"ContainerStarted","Data":"3b13cce73454d642216a03ff1fdc2c7cc8d1956c1b8cd172fa4b9a74afce4be9"}
Sep 10 04:41:50.102379 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:50.102334    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-jcl9d" podStartSLOduration=1.741076288 podStartE2EDuration="4.102318074s" totalImagesPullingTime="2.361241786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:50.101882969 +0000 UTC m=+46.863652150" watchObservedRunningTime="2026-09-10 04:41:50.102318074 +0000 UTC m=+46.864087256"
Sep 10 04:41:50.326344 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:50.326304    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29a6dafa-c02d-4b50-a299-f6b873560b52-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-n2rcz\" (UID: \"29a6dafa-c02d-4b50-a299-f6b873560b52\") " pod="openshift-network-console/networking-console-plugin-79977f9844-n2rcz"
Sep 10 04:41:50.326532 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:50.326439    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 04:41:50.326532 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:50.326500    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29a6dafa-c02d-4b50-a299-f6b873560b52-networking-console-plugin-cert podName:29a6dafa-c02d-4b50-a299-f6b873560b52 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:54.326485448 +0000 UTC m=+51.088254608 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/29a6dafa-c02d-4b50-a299-f6b873560b52-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-n2rcz" (UID: "29a6dafa-c02d-4b50-a299-f6b873560b52") : secret "networking-console-plugin-cert" not found
Sep 10 04:41:50.531808 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:50.531778    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:50.780618 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:50.780574    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:50.968649 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:50.968606    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:51.068391 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:51.068355    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:51.315644 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:51.315556    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:51.416429 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:51.416392    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:51.542925 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:51.542891    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:51.679297 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:51.679207    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:52.461886 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:52.461853    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:52.846182 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:52.846101    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6440e031-173d-40dd-8186-b80bb9fb3fdb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:52.846182 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:52.846139    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6440e031-173d-40dd-8186-b80bb9fb3fdb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:52.846411 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:52.846302    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 04:41:52.846411 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:52.846382    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6440e031-173d-40dd-8186-b80bb9fb3fdb-insights-runtime-extractor-tls podName:6440e031-173d-40dd-8186-b80bb9fb3fdb nodeName:}" failed. No retries permitted until 2026-09-10 04:42:00.846360185 +0000 UTC m=+57.608129367 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6440e031-173d-40dd-8186-b80bb9fb3fdb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-srmv2" (UID: "6440e031-173d-40dd-8186-b80bb9fb3fdb") : secret "insights-runtime-extractor-tls" not found
Sep 10 04:41:52.846698 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:52.846671    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6440e031-173d-40dd-8186-b80bb9fb3fdb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:41:52.870260 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:52.870233    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:52.947278 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:52.947248    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-registry-tls\") pod \"image-registry-5694dd86c5-7rqjv\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") " pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:41:52.947399 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:52.947327    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b748ea1b-814f-474d-9991-2668883967a3-metrics-tls\") pod \"dns-default-mmwjm\" (UID: \"b748ea1b-814f-474d-9991-2668883967a3\") " pod="openshift-dns/dns-default-mmwjm"
Sep 10 04:41:52.947399 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:52.947381    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 04:41:52.947483 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:52.947400    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5694dd86c5-7rqjv: secret "image-registry-tls" not found
Sep 10 04:41:52.947483 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:52.947453    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-registry-tls podName:340229b7-36e5-4ce0-8220-86cf754194e0 nodeName:}" failed. No retries permitted until 2026-09-10 04:42:08.947438773 +0000 UTC m=+65.709207937 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-registry-tls") pod "image-registry-5694dd86c5-7rqjv" (UID: "340229b7-36e5-4ce0-8220-86cf754194e0") : secret "image-registry-tls" not found
Sep 10 04:41:52.947483 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:52.947456    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 04:41:52.947637 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:52.947502    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b748ea1b-814f-474d-9991-2668883967a3-metrics-tls podName:b748ea1b-814f-474d-9991-2668883967a3 nodeName:}" failed. No retries permitted until 2026-09-10 04:42:08.947487966 +0000 UTC m=+65.709257127 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b748ea1b-814f-474d-9991-2668883967a3-metrics-tls") pod "dns-default-mmwjm" (UID: "b748ea1b-814f-474d-9991-2668883967a3") : secret "dns-default-metrics-tls" not found
Sep 10 04:41:53.298768 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:53.298734    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:53.460209 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:53.460168    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:54.067602 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:54.067552    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:54.167609 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:54.167570    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:54.357660 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:54.357577    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29a6dafa-c02d-4b50-a299-f6b873560b52-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-n2rcz\" (UID: \"29a6dafa-c02d-4b50-a299-f6b873560b52\") " pod="openshift-network-console/networking-console-plugin-79977f9844-n2rcz"
Sep 10 04:41:54.357810 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:54.357732    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 04:41:54.357810 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:41:54.357795    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29a6dafa-c02d-4b50-a299-f6b873560b52-networking-console-plugin-cert podName:29a6dafa-c02d-4b50-a299-f6b873560b52 nodeName:}" failed. No retries permitted until 2026-09-10 04:42:02.357780857 +0000 UTC m=+59.119550023 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/29a6dafa-c02d-4b50-a299-f6b873560b52-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-n2rcz" (UID: "29a6dafa-c02d-4b50-a299-f6b873560b52") : secret "networking-console-plugin-cert" not found
Sep 10 04:41:54.564106 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:54.564058    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:54.664388 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:54.664310    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:54.768940 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:54.768889    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:54.867431 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:54.867396    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:55.275650 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:55.275613    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:55.465716 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:55.465678    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:55.531757 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:55.531674    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:55.856955 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:55.856870    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:56.460499 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:56.460467    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:57.063889 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:57.063855    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:57.295322 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:57.295272    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:57.371800 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:57.371717    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:57.658560 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:57.658467    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:57.857611 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:57.857573    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:58.265249 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:58.265216    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:58.863734 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:58.863691    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:59.059003 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:59.058963    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:41:59.686649 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:41:59.686614    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:00.040183 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:00.040147    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:00.141129 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:00.141096    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:00.465897 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:00.465861    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:00.667771 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:00.667741    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:00.910348 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:00.910264    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6440e031-173d-40dd-8186-b80bb9fb3fdb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:42:00.912538 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:00.912511    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6440e031-173d-40dd-8186-b80bb9fb3fdb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srmv2\" (UID: \"6440e031-173d-40dd-8186-b80bb9fb3fdb\") " pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:42:00.918486 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:00.918463    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-srmv2"
Sep 10 04:42:01.059452 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:01.059417    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:01.185800 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:01.185727    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-srmv2"]
Sep 10 04:42:01.189598 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:42:01.189574    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6440e031_173d_40dd_8186_b80bb9fb3fdb.slice/crio-afbdc087baf8df9a51ab99906f1ffcb871f22d0f6dec0cbac66551c9f0414834 WatchSource:0}: Error finding container afbdc087baf8df9a51ab99906f1ffcb871f22d0f6dec0cbac66551c9f0414834: Status 404 returned error can't find the container with id afbdc087baf8df9a51ab99906f1ffcb871f22d0f6dec0cbac66551c9f0414834
Sep 10 04:42:01.467778 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:01.467741    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:01.660718 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:01.660680    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:02.118989 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:02.118953    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-srmv2" event={"ID":"6440e031-173d-40dd-8186-b80bb9fb3fdb","Type":"ContainerStarted","Data":"63b8f784ac631b5c0c0c593e6a389e12843c9c498c422278dbf358b69c95ae8d"}
Sep 10 04:42:02.119367 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:02.119013    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-srmv2" event={"ID":"6440e031-173d-40dd-8186-b80bb9fb3fdb","Type":"ContainerStarted","Data":"afbdc087baf8df9a51ab99906f1ffcb871f22d0f6dec0cbac66551c9f0414834"}
Sep 10 04:42:02.423598 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:02.423522    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29a6dafa-c02d-4b50-a299-f6b873560b52-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-n2rcz\" (UID: \"29a6dafa-c02d-4b50-a299-f6b873560b52\") " pod="openshift-network-console/networking-console-plugin-79977f9844-n2rcz"
Sep 10 04:42:02.425854 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:02.425836    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29a6dafa-c02d-4b50-a299-f6b873560b52-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-n2rcz\" (UID: \"29a6dafa-c02d-4b50-a299-f6b873560b52\") " pod="openshift-network-console/networking-console-plugin-79977f9844-n2rcz"
Sep 10 04:42:02.430722 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:02.430699    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-n2rcz"
Sep 10 04:42:02.742136 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:02.742102    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-n2rcz"]
Sep 10 04:42:02.744940 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:42:02.744914    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a6dafa_c02d_4b50_a299_f6b873560b52.slice/crio-8740cfa51d644f8170f7811c4b9198ac146d56f32fb699d48ec1e24bd1852d95 WatchSource:0}: Error finding container 8740cfa51d644f8170f7811c4b9198ac146d56f32fb699d48ec1e24bd1852d95: Status 404 returned error can't find the container with id 8740cfa51d644f8170f7811c4b9198ac146d56f32fb699d48ec1e24bd1852d95
Sep 10 04:42:03.038980 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:03.038911    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j2hd2"
Sep 10 04:42:03.124476 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:03.124444    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-srmv2" event={"ID":"6440e031-173d-40dd-8186-b80bb9fb3fdb","Type":"ContainerStarted","Data":"2f09ab7179554fe3e99825acd2a095d0d8c43f9b98250ac98003362f5e7f9d27"}
Sep 10 04:42:03.125656 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:03.125631    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-n2rcz" event={"ID":"29a6dafa-c02d-4b50-a299-f6b873560b52","Type":"ContainerStarted","Data":"8740cfa51d644f8170f7811c4b9198ac146d56f32fb699d48ec1e24bd1852d95"}
Sep 10 04:42:03.596405 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:03.596363    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:03.694784 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:03.694745    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:05.134877 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:05.134832    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-srmv2" event={"ID":"6440e031-173d-40dd-8186-b80bb9fb3fdb","Type":"ContainerStarted","Data":"de1256996d40c4d8c93fdbcbee6e6106fca2e0a287708fc475dc6fc823e5d5b5"}
Sep 10 04:42:05.136467 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:05.136439    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-n2rcz" event={"ID":"29a6dafa-c02d-4b50-a299-f6b873560b52","Type":"ContainerStarted","Data":"3ecb536742960e6ae3fc86a57770f341fe486fc7d064e8b1017ff9f8e331cc53"}
Sep 10 04:42:05.153285 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:05.153233    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-srmv2" podStartSLOduration=18.251606321 podStartE2EDuration="21.153217307s" totalImagesPullingTime="2.901610986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:05.152465364 +0000 UTC m=+61.914234547" watchObservedRunningTime="2026-09-10 04:42:05.153217307 +0000 UTC m=+61.914986489"
Sep 10 04:42:05.167819 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:05.167772    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-n2rcz" podStartSLOduration=17.636058777 podStartE2EDuration="19.167756469s" totalImagesPullingTime="1.531697692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:05.167109598 +0000 UTC m=+61.928878783" watchObservedRunningTime="2026-09-10 04:42:05.167756469 +0000 UTC m=+61.929525678"
Sep 10 04:42:06.385187 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.385154    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6db66b458-l5nxz"]
Sep 10 04:42:06.387543 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.387527    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd6986f79-wbrgz"]
Sep 10 04:42:06.387665 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.387647    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6db66b458-l5nxz"
Sep 10 04:42:06.389231 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.389214    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd6986f79-wbrgz"
Sep 10 04:42:06.398312 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.398293    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 04:42:06.398583 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.398570    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 04:42:06.399412 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.399394    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 04:42:06.399599 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.399579    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 10 04:42:06.400678 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.400660    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 04:42:06.403683 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.403665    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-vd8mq\""
Sep 10 04:42:06.409628 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.409605    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"]
Sep 10 04:42:06.411778 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.411484    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"
Sep 10 04:42:06.415421 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.415398    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6db66b458-l5nxz"]
Sep 10 04:42:06.418475 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.418352    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 10 04:42:06.418475 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.418417    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 10 04:42:06.418475 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.418448    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 10 04:42:06.418700 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.418451    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 10 04:42:06.420469 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.420340    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd6986f79-wbrgz"]
Sep 10 04:42:06.425742 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.425722    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5694dd86c5-7rqjv"]
Sep 10 04:42:06.425944 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:42:06.425916    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv" podUID="340229b7-36e5-4ce0-8220-86cf754194e0"
Sep 10 04:42:06.447585 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.447559    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"]
Sep 10 04:42:06.457337 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.457314    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/59a69322-6e43-40a3-9cf2-ee5336ccb359-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9f5757c4f-m5xsb\" (UID: \"59a69322-6e43-40a3-9cf2-ee5336ccb359\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"
Sep 10 04:42:06.457448 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.457344    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sb5p\" (UniqueName: \"kubernetes.io/projected/262a0d23-ed22-407a-beea-3ae234bc61f9-kube-api-access-2sb5p\") pod \"managed-serviceaccount-addon-agent-6db66b458-l5nxz\" (UID: \"262a0d23-ed22-407a-beea-3ae234bc61f9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6db66b458-l5nxz"
Sep 10 04:42:06.457448 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.457380    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e48f7014-3c12-400b-bf50-7a60b9160d1f-klusterlet-config\") pod \"klusterlet-addon-workmgr-7cd6986f79-wbrgz\" (UID: \"e48f7014-3c12-400b-bf50-7a60b9160d1f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd6986f79-wbrgz"
Sep 10 04:42:06.457448 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.457423    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/262a0d23-ed22-407a-beea-3ae234bc61f9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6db66b458-l5nxz\" (UID: \"262a0d23-ed22-407a-beea-3ae234bc61f9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6db66b458-l5nxz"
Sep 10 04:42:06.457562 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.457453    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e48f7014-3c12-400b-bf50-7a60b9160d1f-tmp\") pod \"klusterlet-addon-workmgr-7cd6986f79-wbrgz\" (UID: \"e48f7014-3c12-400b-bf50-7a60b9160d1f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd6986f79-wbrgz"
Sep 10 04:42:06.457562 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.457492    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59a69322-6e43-40a3-9cf2-ee5336ccb359-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9f5757c4f-m5xsb\" (UID: \"59a69322-6e43-40a3-9cf2-ee5336ccb359\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"
Sep 10 04:42:06.457562 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.457530    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5qht\" (UniqueName: \"kubernetes.io/projected/59a69322-6e43-40a3-9cf2-ee5336ccb359-kube-api-access-d5qht\") pod \"cluster-proxy-proxy-agent-9f5757c4f-m5xsb\" (UID: \"59a69322-6e43-40a3-9cf2-ee5336ccb359\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"
Sep 10 04:42:06.457657 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.457562    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/59a69322-6e43-40a3-9cf2-ee5336ccb359-ca\") pod \"cluster-proxy-proxy-agent-9f5757c4f-m5xsb\" (UID: \"59a69322-6e43-40a3-9cf2-ee5336ccb359\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"
Sep 10 04:42:06.457657 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.457580    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tcdk\" (UniqueName: \"kubernetes.io/projected/e48f7014-3c12-400b-bf50-7a60b9160d1f-kube-api-access-7tcdk\") pod \"klusterlet-addon-workmgr-7cd6986f79-wbrgz\" (UID: \"e48f7014-3c12-400b-bf50-7a60b9160d1f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd6986f79-wbrgz"
Sep 10 04:42:06.457657 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.457610    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a69322-6e43-40a3-9cf2-ee5336ccb359-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9f5757c4f-m5xsb\" (UID: \"59a69322-6e43-40a3-9cf2-ee5336ccb359\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"
Sep 10 04:42:06.457657 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.457625    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/59a69322-6e43-40a3-9cf2-ee5336ccb359-hub\") pod \"cluster-proxy-proxy-agent-9f5757c4f-m5xsb\" (UID: \"59a69322-6e43-40a3-9cf2-ee5336ccb359\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"
Sep 10 04:42:06.521122 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.521089    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d9cd496f5-6fkl6"]
Sep 10 04:42:06.523137 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.523121    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.544855 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.544825    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d9cd496f5-6fkl6"]
Sep 10 04:42:06.558052 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.558024    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a69322-6e43-40a3-9cf2-ee5336ccb359-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9f5757c4f-m5xsb\" (UID: \"59a69322-6e43-40a3-9cf2-ee5336ccb359\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"
Sep 10 04:42:06.558222 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.558059    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-image-registry-private-configuration\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.558222 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.558106    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-ca-trust-extracted\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.558222 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.558132    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/59a69322-6e43-40a3-9cf2-ee5336ccb359-hub\") pod \"cluster-proxy-proxy-agent-9f5757c4f-m5xsb\" (UID: \"59a69322-6e43-40a3-9cf2-ee5336ccb359\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"
Sep 10 04:42:06.558222 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.558160    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-bound-sa-token\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.558474 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.558266    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/59a69322-6e43-40a3-9cf2-ee5336ccb359-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9f5757c4f-m5xsb\" (UID: \"59a69322-6e43-40a3-9cf2-ee5336ccb359\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"
Sep 10 04:42:06.558474 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.558421    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:06.558474 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.558445    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sb5p\" (UniqueName: \"kubernetes.io/projected/262a0d23-ed22-407a-beea-3ae234bc61f9-kube-api-access-2sb5p\") pod \"managed-serviceaccount-addon-agent-6db66b458-l5nxz\" (UID: \"262a0d23-ed22-407a-beea-3ae234bc61f9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6db66b458-l5nxz"
Sep 10 04:42:06.558689 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.558487    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-trusted-ca\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.558689 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.558516    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdxgk\" (UniqueName: \"kubernetes.io/projected/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-kube-api-access-gdxgk\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.558689 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.558674    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e48f7014-3c12-400b-bf50-7a60b9160d1f-klusterlet-config\") pod \"klusterlet-addon-workmgr-7cd6986f79-wbrgz\" (UID: \"e48f7014-3c12-400b-bf50-7a60b9160d1f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd6986f79-wbrgz"
Sep 10 04:42:06.558814 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.558734    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/262a0d23-ed22-407a-beea-3ae234bc61f9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6db66b458-l5nxz\" (UID: \"262a0d23-ed22-407a-beea-3ae234bc61f9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6db66b458-l5nxz"
Sep 10 04:42:06.558814 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.558767    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e48f7014-3c12-400b-bf50-7a60b9160d1f-tmp\") pod \"klusterlet-addon-workmgr-7cd6986f79-wbrgz\" (UID: \"e48f7014-3c12-400b-bf50-7a60b9160d1f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd6986f79-wbrgz"
Sep 10 04:42:06.558814 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.558799    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59a69322-6e43-40a3-9cf2-ee5336ccb359-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9f5757c4f-m5xsb\" (UID: \"59a69322-6e43-40a3-9cf2-ee5336ccb359\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"
Sep 10 04:42:06.558969 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.558846    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-registry-tls\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.558969 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.558875    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qht\" (UniqueName: \"kubernetes.io/projected/59a69322-6e43-40a3-9cf2-ee5336ccb359-kube-api-access-d5qht\") pod \"cluster-proxy-proxy-agent-9f5757c4f-m5xsb\" (UID: \"59a69322-6e43-40a3-9cf2-ee5336ccb359\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"
Sep 10 04:42:06.558969 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.558908    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a69322-6e43-40a3-9cf2-ee5336ccb359-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9f5757c4f-m5xsb\" (UID: \"59a69322-6e43-40a3-9cf2-ee5336ccb359\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"
Sep 10 04:42:06.558969 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.558919    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/59a69322-6e43-40a3-9cf2-ee5336ccb359-ca\") pod \"cluster-proxy-proxy-agent-9f5757c4f-m5xsb\" (UID: \"59a69322-6e43-40a3-9cf2-ee5336ccb359\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"
Sep 10 04:42:06.558969 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.558944    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-registry-certificates\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.559192 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.558972    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tcdk\" (UniqueName: \"kubernetes.io/projected/e48f7014-3c12-400b-bf50-7a60b9160d1f-kube-api-access-7tcdk\") pod \"klusterlet-addon-workmgr-7cd6986f79-wbrgz\" (UID: \"e48f7014-3c12-400b-bf50-7a60b9160d1f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd6986f79-wbrgz"
Sep 10 04:42:06.559192 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.559005    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-installation-pull-secrets\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.559300 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.559273    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e48f7014-3c12-400b-bf50-7a60b9160d1f-tmp\") pod \"klusterlet-addon-workmgr-7cd6986f79-wbrgz\" (UID: \"e48f7014-3c12-400b-bf50-7a60b9160d1f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd6986f79-wbrgz"
Sep 10 04:42:06.561006 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.560967    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/59a69322-6e43-40a3-9cf2-ee5336ccb359-hub\") pod \"cluster-proxy-proxy-agent-9f5757c4f-m5xsb\" (UID: \"59a69322-6e43-40a3-9cf2-ee5336ccb359\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"
Sep 10 04:42:06.561144 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.561108    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/59a69322-6e43-40a3-9cf2-ee5336ccb359-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9f5757c4f-m5xsb\" (UID: \"59a69322-6e43-40a3-9cf2-ee5336ccb359\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"
Sep 10 04:42:06.561348 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.561330    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/262a0d23-ed22-407a-beea-3ae234bc61f9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6db66b458-l5nxz\" (UID: \"262a0d23-ed22-407a-beea-3ae234bc61f9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6db66b458-l5nxz"
Sep 10 04:42:06.561603 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.561583    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/59a69322-6e43-40a3-9cf2-ee5336ccb359-ca\") pod \"cluster-proxy-proxy-agent-9f5757c4f-m5xsb\" (UID: \"59a69322-6e43-40a3-9cf2-ee5336ccb359\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"
Sep 10 04:42:06.561900 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.561885    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e48f7014-3c12-400b-bf50-7a60b9160d1f-klusterlet-config\") pod \"klusterlet-addon-workmgr-7cd6986f79-wbrgz\" (UID: \"e48f7014-3c12-400b-bf50-7a60b9160d1f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd6986f79-wbrgz"
Sep 10 04:42:06.562127 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.562106    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59a69322-6e43-40a3-9cf2-ee5336ccb359-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9f5757c4f-m5xsb\" (UID: \"59a69322-6e43-40a3-9cf2-ee5336ccb359\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"
Sep 10 04:42:06.576397 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.576371    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sb5p\" (UniqueName: \"kubernetes.io/projected/262a0d23-ed22-407a-beea-3ae234bc61f9-kube-api-access-2sb5p\") pod \"managed-serviceaccount-addon-agent-6db66b458-l5nxz\" (UID: \"262a0d23-ed22-407a-beea-3ae234bc61f9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6db66b458-l5nxz"
Sep 10 04:42:06.576845 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.576825    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tcdk\" (UniqueName: \"kubernetes.io/projected/e48f7014-3c12-400b-bf50-7a60b9160d1f-kube-api-access-7tcdk\") pod \"klusterlet-addon-workmgr-7cd6986f79-wbrgz\" (UID: \"e48f7014-3c12-400b-bf50-7a60b9160d1f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd6986f79-wbrgz"
Sep 10 04:42:06.577273 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.577254    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5qht\" (UniqueName: \"kubernetes.io/projected/59a69322-6e43-40a3-9cf2-ee5336ccb359-kube-api-access-d5qht\") pod \"cluster-proxy-proxy-agent-9f5757c4f-m5xsb\" (UID: \"59a69322-6e43-40a3-9cf2-ee5336ccb359\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"
Sep 10 04:42:06.642195 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.642121    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:06.660176 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.660145    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-registry-certificates\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.660322 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.660178    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-installation-pull-secrets\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.660322 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.660207    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-image-registry-private-configuration\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.660322 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.660225    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-ca-trust-extracted\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.660442 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.660326    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-bound-sa-token\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.660442 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.660386    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-trusted-ca\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.660442 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.660412    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdxgk\" (UniqueName: \"kubernetes.io/projected/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-kube-api-access-gdxgk\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.660590 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.660509    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-registry-tls\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.660648 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.660588    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-ca-trust-extracted\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.661177 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.661150    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-registry-certificates\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.661416 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.661396    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-trusted-ca\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.662692 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.662668    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-image-registry-private-configuration\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.663154 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.663127    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-installation-pull-secrets\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.663269 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.663186    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-registry-tls\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.673330 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.673310    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-bound-sa-token\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.673444 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.673428    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdxgk\" (UniqueName: \"kubernetes.io/projected/ff2cb19d-c6a1-4a02-9b91-f2d3841ac177-kube-api-access-gdxgk\") pod \"image-registry-7d9cd496f5-6fkl6\" (UID: \"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177\") " pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.715563 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.715535    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6db66b458-l5nxz"
Sep 10 04:42:06.723833 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.723809    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd6986f79-wbrgz"
Sep 10 04:42:06.728825 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.728803    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"
Sep 10 04:42:06.741017 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.740979    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:06.841521 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.841489    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:06.852359 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.852333    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6qv7t\""
Sep 10 04:42:06.860529 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.860507    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:06.939439 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.939405    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:06.998871 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.998839    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6db66b458-l5nxz"]
Sep 10 04:42:06.999827 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:06.999802    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd6986f79-wbrgz"]
Sep 10 04:42:07.002433 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:42:07.002387    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48f7014_3c12_400b_bf50_7a60b9160d1f.slice/crio-8ce585f052577f423c0de7681e77c3de673f6ce2ebe082b2f5de8a2f0830d465 WatchSource:0}: Error finding container 8ce585f052577f423c0de7681e77c3de673f6ce2ebe082b2f5de8a2f0830d465: Status 404 returned error can't find the container with id 8ce585f052577f423c0de7681e77c3de673f6ce2ebe082b2f5de8a2f0830d465
Sep 10 04:42:07.003291 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:42:07.003267    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod262a0d23_ed22_407a_beea_3ae234bc61f9.slice/crio-eabc6f4dbbbc9a837cb36e56f6b0cd6b9c782ec3f60196ade0f0d935a734f6a3 WatchSource:0}: Error finding container eabc6f4dbbbc9a837cb36e56f6b0cd6b9c782ec3f60196ade0f0d935a734f6a3: Status 404 returned error can't find the container with id eabc6f4dbbbc9a837cb36e56f6b0cd6b9c782ec3f60196ade0f0d935a734f6a3
Sep 10 04:42:07.023728 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.023657    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb"]
Sep 10 04:42:07.025581 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:42:07.025556    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59a69322_6e43_40a3_9cf2_ee5336ccb359.slice/crio-bf97f7e5886672c275b1065eb85b6a79c0e0845d1d2dcc0c787bbd76ed3d7ae4 WatchSource:0}: Error finding container bf97f7e5886672c275b1065eb85b6a79c0e0845d1d2dcc0c787bbd76ed3d7ae4: Status 404 returned error can't find the container with id bf97f7e5886672c275b1065eb85b6a79c0e0845d1d2dcc0c787bbd76ed3d7ae4
Sep 10 04:42:07.134771 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.134741    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d9cd496f5-6fkl6"]
Sep 10 04:42:07.137883 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:42:07.137857    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2cb19d_c6a1_4a02_9b91_f2d3841ac177.slice/crio-7f2dd59523c2969062fb583849f61dff75ee70b3b7c5373b9917cbb5c9e4e814 WatchSource:0}: Error finding container 7f2dd59523c2969062fb583849f61dff75ee70b3b7c5373b9917cbb5c9e4e814: Status 404 returned error can't find the container with id 7f2dd59523c2969062fb583849f61dff75ee70b3b7c5373b9917cbb5c9e4e814
Sep 10 04:42:07.143672 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.143566    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6" event={"ID":"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177","Type":"ContainerStarted","Data":"7f2dd59523c2969062fb583849f61dff75ee70b3b7c5373b9917cbb5c9e4e814"}
Sep 10 04:42:07.144791 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.144766    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6db66b458-l5nxz" event={"ID":"262a0d23-ed22-407a-beea-3ae234bc61f9","Type":"ContainerStarted","Data":"eabc6f4dbbbc9a837cb36e56f6b0cd6b9c782ec3f60196ade0f0d935a734f6a3"}
Sep 10 04:42:07.145964 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.145943    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb" event={"ID":"59a69322-6e43-40a3-9cf2-ee5336ccb359","Type":"ContainerStarted","Data":"bf97f7e5886672c275b1065eb85b6a79c0e0845d1d2dcc0c787bbd76ed3d7ae4"}
Sep 10 04:42:07.147110 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.147091    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:42:07.147110 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.147094    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd6986f79-wbrgz" event={"ID":"e48f7014-3c12-400b-bf50-7a60b9160d1f","Type":"ContainerStarted","Data":"8ce585f052577f423c0de7681e77c3de673f6ce2ebe082b2f5de8a2f0830d465"}
Sep 10 04:42:07.152896 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.152877    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:42:07.266292 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.266260    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptnlm\" (UniqueName: \"kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-kube-api-access-ptnlm\") pod \"340229b7-36e5-4ce0-8220-86cf754194e0\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") "
Sep 10 04:42:07.266460 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.266314    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/340229b7-36e5-4ce0-8220-86cf754194e0-ca-trust-extracted\") pod \"340229b7-36e5-4ce0-8220-86cf754194e0\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") "
Sep 10 04:42:07.266460 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.266339    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/340229b7-36e5-4ce0-8220-86cf754194e0-installation-pull-secrets\") pod \"340229b7-36e5-4ce0-8220-86cf754194e0\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") "
Sep 10 04:42:07.266460 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.266363    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/340229b7-36e5-4ce0-8220-86cf754194e0-registry-certificates\") pod \"340229b7-36e5-4ce0-8220-86cf754194e0\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") "
Sep 10 04:42:07.266460 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.266382    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/340229b7-36e5-4ce0-8220-86cf754194e0-trusted-ca\") pod \"340229b7-36e5-4ce0-8220-86cf754194e0\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") "
Sep 10 04:42:07.266460 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.266410    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/340229b7-36e5-4ce0-8220-86cf754194e0-image-registry-private-configuration\") pod \"340229b7-36e5-4ce0-8220-86cf754194e0\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") "
Sep 10 04:42:07.266460 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.266439    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-bound-sa-token\") pod \"340229b7-36e5-4ce0-8220-86cf754194e0\" (UID: \"340229b7-36e5-4ce0-8220-86cf754194e0\") "
Sep 10 04:42:07.266794 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.266743    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340229b7-36e5-4ce0-8220-86cf754194e0-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "340229b7-36e5-4ce0-8220-86cf754194e0" (UID: "340229b7-36e5-4ce0-8220-86cf754194e0"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 04:42:07.267423 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.266916    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/340229b7-36e5-4ce0-8220-86cf754194e0-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "340229b7-36e5-4ce0-8220-86cf754194e0" (UID: "340229b7-36e5-4ce0-8220-86cf754194e0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 04:42:07.267423 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.267106    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/340229b7-36e5-4ce0-8220-86cf754194e0-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "340229b7-36e5-4ce0-8220-86cf754194e0" (UID: "340229b7-36e5-4ce0-8220-86cf754194e0"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 04:42:07.268942 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.268894    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "340229b7-36e5-4ce0-8220-86cf754194e0" (UID: "340229b7-36e5-4ce0-8220-86cf754194e0"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 04:42:07.268942 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.268896    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/340229b7-36e5-4ce0-8220-86cf754194e0-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "340229b7-36e5-4ce0-8220-86cf754194e0" (UID: "340229b7-36e5-4ce0-8220-86cf754194e0"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 04:42:07.269147 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.268968    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-kube-api-access-ptnlm" (OuterVolumeSpecName: "kube-api-access-ptnlm") pod "340229b7-36e5-4ce0-8220-86cf754194e0" (UID: "340229b7-36e5-4ce0-8220-86cf754194e0"). InnerVolumeSpecName "kube-api-access-ptnlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 04:42:07.269147 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.268973    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/340229b7-36e5-4ce0-8220-86cf754194e0-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "340229b7-36e5-4ce0-8220-86cf754194e0" (UID: "340229b7-36e5-4ce0-8220-86cf754194e0"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 04:42:07.367287 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.367251    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/340229b7-36e5-4ce0-8220-86cf754194e0-trusted-ca\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:42:07.367287 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.367281    2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/340229b7-36e5-4ce0-8220-86cf754194e0-image-registry-private-configuration\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:42:07.367287 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.367295    2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-bound-sa-token\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:42:07.367513 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.367309    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptnlm\" (UniqueName: \"kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-kube-api-access-ptnlm\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:42:07.367513 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.367320    2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/340229b7-36e5-4ce0-8220-86cf754194e0-ca-trust-extracted\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:42:07.367513 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.367334    2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/340229b7-36e5-4ce0-8220-86cf754194e0-installation-pull-secrets\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:42:07.367513 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:07.367345    2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/340229b7-36e5-4ce0-8220-86cf754194e0-registry-certificates\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:42:08.036525 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:08.036487    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:08.135981 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:08.135855    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:08.156849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:08.155619    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5694dd86c5-7rqjv"
Sep 10 04:42:08.156849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:08.156579    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6" event={"ID":"ff2cb19d-c6a1-4a02-9b91-f2d3841ac177","Type":"ContainerStarted","Data":"cae1465ae7c2b5bf3232abcc5822dee72aa12ea23778a5f9b8a8e771d7796c6b"}
Sep 10 04:42:08.156849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:08.156628    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:08.197512 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:08.195623    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6" podStartSLOduration=2.195599937 podStartE2EDuration="2.195599937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:08.194120392 +0000 UTC m=+64.955889575" watchObservedRunningTime="2026-09-10 04:42:08.195599937 +0000 UTC m=+64.957369117"
Sep 10 04:42:08.222604 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:08.222529    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5694dd86c5-7rqjv"]
Sep 10 04:42:08.225548 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:08.225487    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5694dd86c5-7rqjv"]
Sep 10 04:42:08.284240 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:08.284204    2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/340229b7-36e5-4ce0-8220-86cf754194e0-registry-tls\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:42:08.990846 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:08.990487    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b748ea1b-814f-474d-9991-2668883967a3-metrics-tls\") pod \"dns-default-mmwjm\" (UID: \"b748ea1b-814f-474d-9991-2668883967a3\") " pod="openshift-dns/dns-default-mmwjm"
Sep 10 04:42:08.994281 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:08.994218    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b748ea1b-814f-474d-9991-2668883967a3-metrics-tls\") pod \"dns-default-mmwjm\" (UID: \"b748ea1b-814f-474d-9991-2668883967a3\") " pod="openshift-dns/dns-default-mmwjm"
Sep 10 04:42:09.205187 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:09.204933    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nj7q4\""
Sep 10 04:42:09.213489 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:09.213158    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mmwjm"
Sep 10 04:42:09.495832 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:09.495745    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs\") pod \"network-metrics-daemon-7krfk\" (UID: \"4bbb6665-dfae-48eb-9f23-f5147f2ccebb\") " pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:42:09.495832 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:09.495801    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret\") pod \"global-pull-secret-syncer-pclmn\" (UID: \"920551bb-5e01-45e3-80cd-87f46761fb29\") " pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:42:09.499397 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:09.499200    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 04:42:09.499648 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:09.499625    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 04:42:09.512117 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:09.512083    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbb6665-dfae-48eb-9f23-f5147f2ccebb-metrics-certs\") pod \"network-metrics-daemon-7krfk\" (UID: \"4bbb6665-dfae-48eb-9f23-f5147f2ccebb\") " pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:42:09.513894 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:09.513845    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/920551bb-5e01-45e3-80cd-87f46761fb29-original-pull-secret\") pod \"global-pull-secret-syncer-pclmn\" (UID: \"920551bb-5e01-45e3-80cd-87f46761fb29\") " pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:42:09.537795 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:09.537760    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mmwjm"]
Sep 10 04:42:09.596996 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:09.596949    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7hpx\" (UniqueName: \"kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx\") pod \"network-check-target-47tk5\" (UID: \"7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8\") " pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:42:09.607241 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:09.607202    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 04:42:09.609897 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:09.609866    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 04:42:09.620885 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:09.620817    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7hpx\" (UniqueName: \"kubernetes.io/projected/7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8-kube-api-access-r7hpx\") pod \"network-check-target-47tk5\" (UID: \"7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8\") " pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:42:09.669977 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:09.669753    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclmn"
Sep 10 04:42:09.678470 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:09.678433    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gx84n\""
Sep 10 04:42:09.683905 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:09.683699    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jksxr\""
Sep 10 04:42:09.686925 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:09.686711    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:42:09.692139 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:09.692099    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7krfk"
Sep 10 04:42:09.864135 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:09.863922    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="340229b7-36e5-4ce0-8220-86cf754194e0" path="/var/lib/kubelet/pods/340229b7-36e5-4ce0-8220-86cf754194e0/volumes"
Sep 10 04:42:10.049820 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:10.049754    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:10.155281 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:10.155136    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:10.255799 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:10.255751    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:10.326792 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:10.326707    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:10.386442 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:10.386406    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:10.406156 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:42:10.406084    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb748ea1b_814f_474d_9991_2668883967a3.slice/crio-2d8cb57f077b52fcfdca16ef15d07f5e3d1cdd0d09177548a36cc1b1db116cd7 WatchSource:0}: Error finding container 2d8cb57f077b52fcfdca16ef15d07f5e3d1cdd0d09177548a36cc1b1db116cd7: Status 404 returned error can't find the container with id 2d8cb57f077b52fcfdca16ef15d07f5e3d1cdd0d09177548a36cc1b1db116cd7
Sep 10 04:42:10.513215 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:10.513175    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:10.723371 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:10.723335    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:10.823582 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:10.823534    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:10.926773 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:10.926726    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:11.108814 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:11.108725    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:11.171518 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:11.171487    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mmwjm" event={"ID":"b748ea1b-814f-474d-9991-2668883967a3","Type":"ContainerStarted","Data":"2d8cb57f077b52fcfdca16ef15d07f5e3d1cdd0d09177548a36cc1b1db116cd7"}
Sep 10 04:42:11.180595 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:11.180562    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:11.313901 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:11.313855    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:11.411256 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:11.411173    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:11.538221 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:11.538187    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:11.598370 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:11.598329    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:11.743371 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:11.743334    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:11.804798 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:11.804761    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:11.934393 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:11.934349    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:12.318135 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:12.318100    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:12.879436 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:12.879401    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:14.217535 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:14.217449    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:14.227416 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:14.227384    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pclmn"]
Sep 10 04:42:14.294656 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:14.294616    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:14.352673 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:14.352631    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:14.424814 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:42:14.424614    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920551bb_5e01_45e3_80cd_87f46761fb29.slice/crio-81e43e90632ca9503387cc53a7a52d2796c3420cf9ca5dfa6c1acab50a631634 WatchSource:0}: Error finding container 81e43e90632ca9503387cc53a7a52d2796c3420cf9ca5dfa6c1acab50a631634: Status 404 returned error can't find the container with id 81e43e90632ca9503387cc53a7a52d2796c3420cf9ca5dfa6c1acab50a631634
Sep 10 04:42:14.452759 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:14.452720    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:14.692718 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:14.692691    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7krfk"]
Sep 10 04:42:14.697253 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:42:14.697163    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bbb6665_dfae_48eb_9f23_f5147f2ccebb.slice/crio-15bd7611dce8d74a0c13088a807d936b1d84e7e2af42f4a589a5d8481bc27541 WatchSource:0}: Error finding container 15bd7611dce8d74a0c13088a807d936b1d84e7e2af42f4a589a5d8481bc27541: Status 404 returned error can't find the container with id 15bd7611dce8d74a0c13088a807d936b1d84e7e2af42f4a589a5d8481bc27541
Sep 10 04:42:14.697748 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:14.697725    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-47tk5"]
Sep 10 04:42:14.703183 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:42:14.703159    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a5ba931_6188_40d5_a5f4_4d50c3b9c8a8.slice/crio-bdc87524677f4f0c7dc3d8502edec084ce6bf718cacc4040c1a0fb1cd3b541b4 WatchSource:0}: Error finding container bdc87524677f4f0c7dc3d8502edec084ce6bf718cacc4040c1a0fb1cd3b541b4: Status 404 returned error can't find the container with id bdc87524677f4f0c7dc3d8502edec084ce6bf718cacc4040c1a0fb1cd3b541b4
Sep 10 04:42:15.189013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:15.188974    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6db66b458-l5nxz" event={"ID":"262a0d23-ed22-407a-beea-3ae234bc61f9","Type":"ContainerStarted","Data":"ba099a8849a810dceb5d2e5d1421e89fdf67c6869686fff19f8817ff44a4064d"}
Sep 10 04:42:15.190510 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:15.190482    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-47tk5" event={"ID":"7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8","Type":"ContainerStarted","Data":"bdc87524677f4f0c7dc3d8502edec084ce6bf718cacc4040c1a0fb1cd3b541b4"}
Sep 10 04:42:15.192001 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:15.191973    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pclmn" event={"ID":"920551bb-5e01-45e3-80cd-87f46761fb29","Type":"ContainerStarted","Data":"81e43e90632ca9503387cc53a7a52d2796c3420cf9ca5dfa6c1acab50a631634"}
Sep 10 04:42:15.195799 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:15.195774    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb" event={"ID":"59a69322-6e43-40a3-9cf2-ee5336ccb359","Type":"ContainerStarted","Data":"e4a10532ce3f53989417432937866a9cbd88754464ed7ea7fa34d14547787afc"}
Sep 10 04:42:15.195903 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:15.195809    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb" event={"ID":"59a69322-6e43-40a3-9cf2-ee5336ccb359","Type":"ContainerStarted","Data":"3fd22d905a4d486a603121166a1115cad3931f95efe7008966ce140f17f3eb5b"}
Sep 10 04:42:15.195903 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:15.195825    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb" event={"ID":"59a69322-6e43-40a3-9cf2-ee5336ccb359","Type":"ContainerStarted","Data":"7927334477ec0e11faf586149413447cbd4d4c06619bba8a58776c9e0b72c565"}
Sep 10 04:42:15.197948 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:15.197926    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd6986f79-wbrgz" event={"ID":"e48f7014-3c12-400b-bf50-7a60b9160d1f","Type":"ContainerStarted","Data":"c6cbb090ca7826cb82bd5432b8c42b15de12f3c24c8230aad4cd27fea6eb8ddb"}
Sep 10 04:42:15.198120 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:15.198100    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd6986f79-wbrgz"
Sep 10 04:42:15.200028 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:15.200008    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd6986f79-wbrgz"
Sep 10 04:42:15.200422 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:15.200401    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mmwjm" event={"ID":"b748ea1b-814f-474d-9991-2668883967a3","Type":"ContainerStarted","Data":"7fa5cccf79ab2c13ee010690a19b46e9ce2fdbed48b61a899d52ae28e0a51263"}
Sep 10 04:42:15.200502 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:15.200448    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mmwjm" event={"ID":"b748ea1b-814f-474d-9991-2668883967a3","Type":"ContainerStarted","Data":"0d022aa1cd9158c6b5cbca095ad2c3557b2ab60056bdb700055e6d9e692859d0"}
Sep 10 04:42:15.200581 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:15.200501    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mmwjm"
Sep 10 04:42:15.202027 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:15.202002    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7krfk" event={"ID":"4bbb6665-dfae-48eb-9f23-f5147f2ccebb","Type":"ContainerStarted","Data":"15bd7611dce8d74a0c13088a807d936b1d84e7e2af42f4a589a5d8481bc27541"}
Sep 10 04:42:15.203982 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:15.203945    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6db66b458-l5nxz" podStartSLOduration=1.784083402 podStartE2EDuration="9.203933663s" totalImagesPullingTime="7.419850261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:15.203860677 +0000 UTC m=+71.965629860" watchObservedRunningTime="2026-09-10 04:42:15.203933663 +0000 UTC m=+71.965702845"
Sep 10 04:42:15.222677 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:15.222630    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd6986f79-wbrgz" podStartSLOduration=1.801955169 podStartE2EDuration="9.222614624s" totalImagesPullingTime="7.420659455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:15.221211501 +0000 UTC m=+71.982980685" watchObservedRunningTime="2026-09-10 04:42:15.222614624 +0000 UTC m=+71.984383796"
Sep 10 04:42:15.241655 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:15.241583    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f5757c4f-m5xsb" podStartSLOduration=1.8436172640000001 podStartE2EDuration="9.241566539s" totalImagesPullingTime="7.397949275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:15.241014023 +0000 UTC m=+72.002783206" watchObservedRunningTime="2026-09-10 04:42:15.241566539 +0000 UTC m=+72.003335721"
Sep 10 04:42:15.260815 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:15.260748    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mmwjm" podStartSLOduration=34.20851227 podStartE2EDuration="38.260728411s" totalImagesPullingTime="4.052216141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:15.259475251 +0000 UTC m=+72.021244434" watchObservedRunningTime="2026-09-10 04:42:15.260728411 +0000 UTC m=+72.022497594"
Sep 10 04:42:16.207834 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:16.207788    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7krfk" event={"ID":"4bbb6665-dfae-48eb-9f23-f5147f2ccebb","Type":"ContainerStarted","Data":"ec3e88f74a4508aed75c5e50a01f3b9d0571a1b74283c84dd4af6aa882905fe8"}
Sep 10 04:42:16.208005 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:16.207850    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7krfk" event={"ID":"4bbb6665-dfae-48eb-9f23-f5147f2ccebb","Type":"ContainerStarted","Data":"67c828ec6190b6c96b1f209af38a9f843faf9dcb043d925a56081df9509b81b6"}
Sep 10 04:42:16.226825 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:16.226767    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7krfk" podStartSLOduration=71.20384302 podStartE2EDuration="1m12.226743533s" totalImagesPullingTime="1.022900513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:16.225520778 +0000 UTC m=+72.987289960" watchObservedRunningTime="2026-09-10 04:42:16.226743533 +0000 UTC m=+72.988512716"
Sep 10 04:42:16.346601 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:16.346537    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:16.447782 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:16.447738    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:16.641059 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:16.640959    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:19.132396 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.132360    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:19.195719 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.195671    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:19.380109 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.380058    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:19.642192 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.642152    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:19.726439 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.726401    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bbztc"]
Sep 10 04:42:19.730234 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.730215    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.734451 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.734426    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 04:42:19.735787 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.735761    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 04:42:19.735787 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.735773    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9hw97\""
Sep 10 04:42:19.736096 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.736058    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 04:42:19.736289 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.736275    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 04:42:19.736460 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.736448    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 04:42:19.736643 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.736620    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 04:42:19.783925 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.783847    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:19.882849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.882816    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.882849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.882853    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-sys\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.883030 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.882881    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77fjs\" (UniqueName: \"kubernetes.io/projected/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-kube-api-access-77fjs\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.883030 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.882908    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-node-exporter-accelerators-collector-config\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.883030 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.882924    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-metrics-client-ca\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.883030 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.882942    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-root\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.883030 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.882972    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-node-exporter-textfile\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.883030 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.882996    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-node-exporter-tls\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.883030 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.883022    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-node-exporter-wtmp\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.895608 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.895585    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:19.984354 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.984315    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.984354 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.984353    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-sys\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.984585 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.984380    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77fjs\" (UniqueName: \"kubernetes.io/projected/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-kube-api-access-77fjs\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.984585 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.984399    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-node-exporter-accelerators-collector-config\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.984585 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.984415    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-metrics-client-ca\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.984585 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.984433    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-root\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.984585 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.984451    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-sys\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.984585 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.984458    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-node-exporter-textfile\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.984585 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.984521    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-node-exporter-tls\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.984585 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.984517    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-root\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.984585 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.984548    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-node-exporter-wtmp\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.985003 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.984726    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-node-exporter-wtmp\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.985003 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.984916    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-node-exporter-textfile\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.985153 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.985097    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-node-exporter-accelerators-collector-config\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.985210 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.985176    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-metrics-client-ca\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.986615 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.986596    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.986793 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.986773    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-node-exporter-tls\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:19.997036 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:19.997006    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77fjs\" (UniqueName: \"kubernetes.io/projected/dc941e6f-c3da-4ed4-b77a-3d3157936f7f-kube-api-access-77fjs\") pod \"node-exporter-bbztc\" (UID: \"dc941e6f-c3da-4ed4-b77a-3d3157936f7f\") " pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:20.040456 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:20.040376    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:20.042238 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:20.042223    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bbztc"
Sep 10 04:42:20.053539 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:42:20.053516    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc941e6f_c3da_4ed4_b77a_3d3157936f7f.slice/crio-a6d4c63d02b89dcd5263e9b259caef8e52d29e038b45741476d2393537f5a47f WatchSource:0}: Error finding container a6d4c63d02b89dcd5263e9b259caef8e52d29e038b45741476d2393537f5a47f: Status 404 returned error can't find the container with id a6d4c63d02b89dcd5263e9b259caef8e52d29e038b45741476d2393537f5a47f
Sep 10 04:42:20.141230 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:20.141198    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:20.225906 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:20.225871    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-47tk5" event={"ID":"7a5ba931-6188-40d5-a5f4-4d50c3b9c8a8","Type":"ContainerStarted","Data":"a514805fd4ac7a4c575963a176777fa27688cc806bb351fe26dbac1d1d0271e8"}
Sep 10 04:42:20.226099 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:20.226025    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:42:20.227229 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:20.227195    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bbztc" event={"ID":"dc941e6f-c3da-4ed4-b77a-3d3157936f7f","Type":"ContainerStarted","Data":"a6d4c63d02b89dcd5263e9b259caef8e52d29e038b45741476d2393537f5a47f"}
Sep 10 04:42:20.228623 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:20.228597    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pclmn" event={"ID":"920551bb-5e01-45e3-80cd-87f46761fb29","Type":"ContainerStarted","Data":"031446a61377b3df117d9030065ac4aad048d1ad8f572559524aec4071215d88"}
Sep 10 04:42:20.242553 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:20.242508    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-47tk5" podStartSLOduration=72.439088044 podStartE2EDuration="1m17.24249442s" totalImagesPullingTime="4.803406376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:20.241956343 +0000 UTC m=+77.003725536" watchObservedRunningTime="2026-09-10 04:42:20.24249442 +0000 UTC m=+77.004263602"
Sep 10 04:42:20.256897 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:20.256750    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pclmn" podStartSLOduration=71.170497236 podStartE2EDuration="1m16.25673641s" totalImagesPullingTime="5.086239174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:20.256497685 +0000 UTC m=+77.018266867" watchObservedRunningTime="2026-09-10 04:42:20.25673641 +0000 UTC m=+77.018505592"
Sep 10 04:42:20.363048 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:20.362962    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:20.515144 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:20.515113    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:20.607918 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:20.607868    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:20.708220 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:20.708169    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:20.786275 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:20.786225    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:20.878744 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:20.878709    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:21.233659 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:21.233609    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bbztc" event={"ID":"dc941e6f-c3da-4ed4-b77a-3d3157936f7f","Type":"ContainerStarted","Data":"d87b384a612fd06f914a72c2b7f4258801520cb6b7608bf662982e54cf47e868"}
Sep 10 04:42:22.239649 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:22.239613    2575 generic.go:356] "Generic (PLEG): container finished" podID="dc941e6f-c3da-4ed4-b77a-3d3157936f7f" containerID="d87b384a612fd06f914a72c2b7f4258801520cb6b7608bf662982e54cf47e868" exitCode=0
Sep 10 04:42:22.240060 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:22.239701    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bbztc" event={"ID":"dc941e6f-c3da-4ed4-b77a-3d3157936f7f","Type":"ContainerDied","Data":"d87b384a612fd06f914a72c2b7f4258801520cb6b7608bf662982e54cf47e868"}
Sep 10 04:42:22.588226 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:22.588137    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:22.649781 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:22.649750    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:22.845733 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:22.845649    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:22.946261 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:22.946226    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:23.245844 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:23.245812    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bbztc" event={"ID":"dc941e6f-c3da-4ed4-b77a-3d3157936f7f","Type":"ContainerStarted","Data":"9288bdab9a77bd218f663e9ad37678c74a1c8196ab6e3de6b36c3b7d2bcc54e1"}
Sep 10 04:42:23.246272 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:23.245894    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bbztc" event={"ID":"dc941e6f-c3da-4ed4-b77a-3d3157936f7f","Type":"ContainerStarted","Data":"2cbfaf3a757dcfe5cc5362e890c133b2ff3760fd5418ad77580375453d215495"}
Sep 10 04:42:23.265601 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:23.265553    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bbztc" podStartSLOduration=3.111769769 podStartE2EDuration="4.265539293s" totalImagesPullingTime="1.089153076s" totalInitContainerRuntime="64.616448ms" isStatefulPod=true podCreationTimestamp="2026-09-10 04:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:23.264150624 +0000 UTC m=+80.025919800" watchObservedRunningTime="2026-09-10 04:42:23.265539293 +0000 UTC m=+80.027308474"
Sep 10 04:42:24.402827 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:24.402786    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:24.489757 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:24.489725    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:24.656632 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:24.656540    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:25.210114 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:25.210086    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mmwjm"
Sep 10 04:42:25.881544 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:25.881408    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:26.864582 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:26.864545    2575 patch_prober.go:28] interesting pod/image-registry-7d9cd496f5-6fkl6 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 10 04:42:26.864822 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:26.864596    2575 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6" podUID="ff2cb19d-c6a1-4a02-9b91-f2d3841ac177" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 04:42:27.191913 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:27.191827    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:27.292888 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:27.292849    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:27.348349 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:27.348313    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:29.166970 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:29.166942    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d9cd496f5-6fkl6"
Sep 10 04:42:29.384801 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:29.384766    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:29.476562 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:29.476528    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:30.312359 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:30.312321    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:30.399561 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:30.399526    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:32.544754 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:32.544712    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:32.649724 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:32.649688    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:33.074631 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:33.074597    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:33.177649 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:33.177595    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:35.407482 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:35.407443    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:35.465789 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:35.465747    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:38.143577 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:38.143537    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:38.246714 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:38.246678    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:39.322822 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:39.322792    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:39.423221 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:39.423183    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:40.458544 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:40.458491    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:40.563756 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:40.563720    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:40.659825 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:40.659790    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:42.159585 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:42.159540    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:42.259239 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:42.259203    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:42.820400 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:42.820368    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:42.922040 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:42.922000    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:50.393169 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:50.393133    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:50.453679 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:50.453636    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:42:51.236789 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:42:51.236758    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-47tk5"
Sep 10 04:43:05.471055 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:43:05.471017    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:43:05.528367 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:43:05.528338    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:43:07.177434 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:43:07.177395    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:43:38.855598 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:43:38.855556    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:43:39.084848 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:43:39.084813    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:43:48.736314 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:43:48.736273    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:43:48.880544 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:43:48.880505    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:43:48.981029 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:43:48.980993    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:43:49.478034 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:43:49.477996    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:43:49.528001 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:43:49.527967    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:43:54.515591 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:43:54.515549    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:43:59.774484 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:43:59.774399    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:43:59.874845 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:43:59.874810    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:43:59.933377 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:43:59.933339    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:44:07.126419 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:44:07.126379    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:44:40.464483 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:44:40.464452    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:44:54.094758 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:44:54.094719    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:44:54.144189 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:44:54.144148    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:45:06.213438 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:45:06.213390    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:45:06.272099 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:45:06.272037    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:45:09.316860 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:45:09.316822    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:45:09.366337 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:45:09.366299    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:45:20.071300 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:45:20.071253    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:45:20.170575 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:45:20.170539    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:45:20.224793 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:45:20.224759    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:45:20.404718 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:45:20.404620    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:45:20.502335 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:45:20.502304    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:46:03.659042 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:46:03.659015    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2hd2_9d909143-4c70-4fe1-900a-3cd520db32e5/ovn-acl-logging/0.log"
Sep 10 04:46:03.662032 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:46:03.662015    2575 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 04:46:03.773272 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:46:03.773245    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2hd2_9d909143-4c70-4fe1-900a-3cd520db32e5/ovn-acl-logging/0.log"
Sep 10 04:46:18.646815 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:46:18.646769    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:46:19.883363 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:46:19.883324    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:46:45.139914 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:46:45.139880    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:46:46.528385 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:46:46.528347    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:46:48.757562 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:46:48.757526    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:46:48.857674 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:46:48.857631    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:46:49.765998 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:46:49.765963    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:46:49.867748 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:46:49.867712    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:46:50.296558 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:46:50.296522    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:46:50.401760 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:46:50.401726    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:46:52.492623 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:46:52.492586    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:46:52.897023 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:46:52.896932    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:46:55.757930 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:46:55.757842    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:46:55.854915 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:46:55.854881    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:46:57.775668 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:46:57.775634    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:46:58.373212 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:46:58.373175    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:00.893908 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:00.893871    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:01.764665 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:01.764616    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:01.861755 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:01.861715    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:08.599988 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:08.599950    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:08.655199 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:08.655162    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:10.660272 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:10.660238    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:26.390809 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:26.390770    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:26.482764 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:26.482726    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:26.965359 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:26.965308    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:27.196559 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.196524    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:27.296801 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.296710    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:27.429995 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.429963    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq"]
Sep 10 04:47:27.432614 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.432593    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq"
Sep 10 04:47:27.435597 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.435576    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 04:47:27.436487 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.436465    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 04:47:27.436595 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.436520    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 04:47:27.436659 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.436601    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 04:47:27.436659 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.436612    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dzbdb\""
Sep 10 04:47:27.446979 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.446957    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq"]
Sep 10 04:47:27.519263 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.519230    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8ee33fc9-f1e7-4b32-af9f-d3a724d2760e-manifests\") pod \"opendatahub-operator-controller-manager-c57f4866c-qztwq\" (UID: \"8ee33fc9-f1e7-4b32-af9f-d3a724d2760e\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq"
Sep 10 04:47:27.519443 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.519278    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ee33fc9-f1e7-4b32-af9f-d3a724d2760e-webhook-cert\") pod \"opendatahub-operator-controller-manager-c57f4866c-qztwq\" (UID: \"8ee33fc9-f1e7-4b32-af9f-d3a724d2760e\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq"
Sep 10 04:47:27.519443 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.519313    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ee33fc9-f1e7-4b32-af9f-d3a724d2760e-tmp\") pod \"opendatahub-operator-controller-manager-c57f4866c-qztwq\" (UID: \"8ee33fc9-f1e7-4b32-af9f-d3a724d2760e\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq"
Sep 10 04:47:27.519443 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.519358    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8ee33fc9-f1e7-4b32-af9f-d3a724d2760e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c57f4866c-qztwq\" (UID: \"8ee33fc9-f1e7-4b32-af9f-d3a724d2760e\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq"
Sep 10 04:47:27.519443 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.519412    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh8h6\" (UniqueName: \"kubernetes.io/projected/8ee33fc9-f1e7-4b32-af9f-d3a724d2760e-kube-api-access-dh8h6\") pod \"opendatahub-operator-controller-manager-c57f4866c-qztwq\" (UID: \"8ee33fc9-f1e7-4b32-af9f-d3a724d2760e\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq"
Sep 10 04:47:27.547649 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.547567    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:27.619750 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.619715    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ee33fc9-f1e7-4b32-af9f-d3a724d2760e-webhook-cert\") pod \"opendatahub-operator-controller-manager-c57f4866c-qztwq\" (UID: \"8ee33fc9-f1e7-4b32-af9f-d3a724d2760e\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq"
Sep 10 04:47:27.619750 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.619757    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ee33fc9-f1e7-4b32-af9f-d3a724d2760e-tmp\") pod \"opendatahub-operator-controller-manager-c57f4866c-qztwq\" (UID: \"8ee33fc9-f1e7-4b32-af9f-d3a724d2760e\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq"
Sep 10 04:47:27.620001 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.619776    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8ee33fc9-f1e7-4b32-af9f-d3a724d2760e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c57f4866c-qztwq\" (UID: \"8ee33fc9-f1e7-4b32-af9f-d3a724d2760e\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq"
Sep 10 04:47:27.620001 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.619810    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh8h6\" (UniqueName: \"kubernetes.io/projected/8ee33fc9-f1e7-4b32-af9f-d3a724d2760e-kube-api-access-dh8h6\") pod \"opendatahub-operator-controller-manager-c57f4866c-qztwq\" (UID: \"8ee33fc9-f1e7-4b32-af9f-d3a724d2760e\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq"
Sep 10 04:47:27.620001 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.619851    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8ee33fc9-f1e7-4b32-af9f-d3a724d2760e-manifests\") pod \"opendatahub-operator-controller-manager-c57f4866c-qztwq\" (UID: \"8ee33fc9-f1e7-4b32-af9f-d3a724d2760e\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq"
Sep 10 04:47:27.620290 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.620268    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8ee33fc9-f1e7-4b32-af9f-d3a724d2760e-manifests\") pod \"opendatahub-operator-controller-manager-c57f4866c-qztwq\" (UID: \"8ee33fc9-f1e7-4b32-af9f-d3a724d2760e\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq"
Sep 10 04:47:27.622083 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.622045    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ee33fc9-f1e7-4b32-af9f-d3a724d2760e-tmp\") pod \"opendatahub-operator-controller-manager-c57f4866c-qztwq\" (UID: \"8ee33fc9-f1e7-4b32-af9f-d3a724d2760e\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq"
Sep 10 04:47:27.622701 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.622681    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8ee33fc9-f1e7-4b32-af9f-d3a724d2760e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c57f4866c-qztwq\" (UID: \"8ee33fc9-f1e7-4b32-af9f-d3a724d2760e\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq"
Sep 10 04:47:27.622797 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.622717    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ee33fc9-f1e7-4b32-af9f-d3a724d2760e-webhook-cert\") pod \"opendatahub-operator-controller-manager-c57f4866c-qztwq\" (UID: \"8ee33fc9-f1e7-4b32-af9f-d3a724d2760e\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq"
Sep 10 04:47:27.630284 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.630262    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh8h6\" (UniqueName: \"kubernetes.io/projected/8ee33fc9-f1e7-4b32-af9f-d3a724d2760e-kube-api-access-dh8h6\") pod \"opendatahub-operator-controller-manager-c57f4866c-qztwq\" (UID: \"8ee33fc9-f1e7-4b32-af9f-d3a724d2760e\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq"
Sep 10 04:47:27.648416 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.648390    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:27.742680 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.742645    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq"
Sep 10 04:47:27.748271 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.748239    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:27.853850 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.853766    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:27.951961 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:27.951926    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:28.017321 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:28.017274    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq"]
Sep 10 04:47:28.021358 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:47:28.021336    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee33fc9_f1e7_4b32_af9f_d3a724d2760e.slice/crio-37589b0e8fe6a442bf10e71fdc9b82aeb39a12a6e2d33649190bb11ae11eb32f WatchSource:0}: Error finding container 37589b0e8fe6a442bf10e71fdc9b82aeb39a12a6e2d33649190bb11ae11eb32f: Status 404 returned error can't find the container with id 37589b0e8fe6a442bf10e71fdc9b82aeb39a12a6e2d33649190bb11ae11eb32f
Sep 10 04:47:28.023136 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:28.023116    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 04:47:28.052809 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:28.052784    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:28.267419 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:28.267388    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq" event={"ID":"8ee33fc9-f1e7-4b32-af9f-d3a724d2760e","Type":"ContainerStarted","Data":"37589b0e8fe6a442bf10e71fdc9b82aeb39a12a6e2d33649190bb11ae11eb32f"}
Sep 10 04:47:29.948250 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:29.948199    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:30.051377 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:30.051341    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:30.160746 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:30.160700    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:30.671641 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:30.671591    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:30.775640 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:30.775601    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:30.905126 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:30.905092    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:31.015657 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:31.015592    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:31.281519 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:31.281429    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq" event={"ID":"8ee33fc9-f1e7-4b32-af9f-d3a724d2760e","Type":"ContainerStarted","Data":"275b1b049ac8a01f9147f4f2448031908f5517332f26b8f7dc6e1cd23f2fe7f9"}
Sep 10 04:47:32.286912 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:32.286878    2575 generic.go:356] "Generic (PLEG): container finished" podID="8ee33fc9-f1e7-4b32-af9f-d3a724d2760e" containerID="275b1b049ac8a01f9147f4f2448031908f5517332f26b8f7dc6e1cd23f2fe7f9" exitCode=0
Sep 10 04:47:32.287440 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:32.286935    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq" event={"ID":"8ee33fc9-f1e7-4b32-af9f-d3a724d2760e","Type":"ContainerDied","Data":"275b1b049ac8a01f9147f4f2448031908f5517332f26b8f7dc6e1cd23f2fe7f9"}
Sep 10 04:47:33.292929 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:33.292892    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq" event={"ID":"8ee33fc9-f1e7-4b32-af9f-d3a724d2760e","Type":"ContainerStarted","Data":"0f86b55926ebf5a95322f7e1a3f3c90d2fb275de70123b4d4bd3a9b6e0986590"}
Sep 10 04:47:33.293337 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:33.293000    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq"
Sep 10 04:47:33.316582 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:33.316535    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq" podStartSLOduration=1.932131847 podStartE2EDuration="6.316518021s" totalImagesPullingTime="3.345009699s" totalInitContainerRuntime="1.039376475s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:47:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:47:33.314887353 +0000 UTC m=+390.076656535" watchObservedRunningTime="2026-09-10 04:47:33.316518021 +0000 UTC m=+390.078287203"
Sep 10 04:47:39.556936 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:39.556903    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:39.677122 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:39.677083    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:40.733236 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:40.733204    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:40.833462 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:40.833426    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:40.934658 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:40.934605    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:44.299362 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:44.299325    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-qztwq"
Sep 10 04:47:46.761353 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:46.761319    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:46.878359 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:46.878318    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:46.981937 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:46.981901    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:47.046665 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:47.046573    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b5848dcdb-f95p8"]
Sep 10 04:47:47.055938 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:47.055916    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b5848dcdb-f95p8"
Sep 10 04:47:47.058911 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:47.058866    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 04:47:47.059918 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:47.059897    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 04:47:47.060019 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:47.059915    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 04:47:47.060019 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:47.059932    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 04:47:47.060019 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:47.059943    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hr9th\""
Sep 10 04:47:47.063010 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:47.062990    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b5848dcdb-f95p8"]
Sep 10 04:47:47.133237 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:47.133196    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:47.171353 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:47.171328    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78923cab-9d82-4bf4-b689-ac5773511687-tls-certs\") pod \"kube-auth-proxy-b5848dcdb-f95p8\" (UID: \"78923cab-9d82-4bf4-b689-ac5773511687\") " pod="openshift-ingress/kube-auth-proxy-b5848dcdb-f95p8"
Sep 10 04:47:47.171465 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:47.171389    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjcvm\" (UniqueName: \"kubernetes.io/projected/78923cab-9d82-4bf4-b689-ac5773511687-kube-api-access-zjcvm\") pod \"kube-auth-proxy-b5848dcdb-f95p8\" (UID: \"78923cab-9d82-4bf4-b689-ac5773511687\") " pod="openshift-ingress/kube-auth-proxy-b5848dcdb-f95p8"
Sep 10 04:47:47.171465 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:47.171412    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78923cab-9d82-4bf4-b689-ac5773511687-tmp\") pod \"kube-auth-proxy-b5848dcdb-f95p8\" (UID: \"78923cab-9d82-4bf4-b689-ac5773511687\") " pod="openshift-ingress/kube-auth-proxy-b5848dcdb-f95p8"
Sep 10 04:47:47.272329 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:47.272299    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78923cab-9d82-4bf4-b689-ac5773511687-tls-certs\") pod \"kube-auth-proxy-b5848dcdb-f95p8\" (UID: \"78923cab-9d82-4bf4-b689-ac5773511687\") " pod="openshift-ingress/kube-auth-proxy-b5848dcdb-f95p8"
Sep 10 04:47:47.272459 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:47.272358    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjcvm\" (UniqueName: \"kubernetes.io/projected/78923cab-9d82-4bf4-b689-ac5773511687-kube-api-access-zjcvm\") pod \"kube-auth-proxy-b5848dcdb-f95p8\" (UID: \"78923cab-9d82-4bf4-b689-ac5773511687\") " pod="openshift-ingress/kube-auth-proxy-b5848dcdb-f95p8"
Sep 10 04:47:47.272459 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:47.272379    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78923cab-9d82-4bf4-b689-ac5773511687-tmp\") pod \"kube-auth-proxy-b5848dcdb-f95p8\" (UID: \"78923cab-9d82-4bf4-b689-ac5773511687\") " pod="openshift-ingress/kube-auth-proxy-b5848dcdb-f95p8"
Sep 10 04:47:47.274613 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:47.274587    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78923cab-9d82-4bf4-b689-ac5773511687-tmp\") pod \"kube-auth-proxy-b5848dcdb-f95p8\" (UID: \"78923cab-9d82-4bf4-b689-ac5773511687\") " pod="openshift-ingress/kube-auth-proxy-b5848dcdb-f95p8"
Sep 10 04:47:47.274827 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:47.274805    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78923cab-9d82-4bf4-b689-ac5773511687-tls-certs\") pod \"kube-auth-proxy-b5848dcdb-f95p8\" (UID: \"78923cab-9d82-4bf4-b689-ac5773511687\") " pod="openshift-ingress/kube-auth-proxy-b5848dcdb-f95p8"
Sep 10 04:47:47.281635 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:47.281608    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjcvm\" (UniqueName: \"kubernetes.io/projected/78923cab-9d82-4bf4-b689-ac5773511687-kube-api-access-zjcvm\") pod \"kube-auth-proxy-b5848dcdb-f95p8\" (UID: \"78923cab-9d82-4bf4-b689-ac5773511687\") " pod="openshift-ingress/kube-auth-proxy-b5848dcdb-f95p8"
Sep 10 04:47:47.367325 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:47.367264    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b5848dcdb-f95p8"
Sep 10 04:47:47.643981 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:47.643956    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b5848dcdb-f95p8"]
Sep 10 04:47:47.646387 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:47:47.646361    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78923cab_9d82_4bf4_b689_ac5773511687.slice/crio-5a17c782d5554082410e5a75ded064785faa89eece6fc4c1f5ba78b042c8e6b3 WatchSource:0}: Error finding container 5a17c782d5554082410e5a75ded064785faa89eece6fc4c1f5ba78b042c8e6b3: Status 404 returned error can't find the container with id 5a17c782d5554082410e5a75ded064785faa89eece6fc4c1f5ba78b042c8e6b3
Sep 10 04:47:48.240197 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:48.240157    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:48.297636 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:48.297570    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:48.352548 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:48.352508    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b5848dcdb-f95p8" event={"ID":"78923cab-9d82-4bf4-b689-ac5773511687","Type":"ContainerStarted","Data":"5a17c782d5554082410e5a75ded064785faa89eece6fc4c1f5ba78b042c8e6b3"}
Sep 10 04:47:49.102043 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:49.102005    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:50.361233 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:50.361200    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b5848dcdb-f95p8" event={"ID":"78923cab-9d82-4bf4-b689-ac5773511687","Type":"ContainerStarted","Data":"4e5e4fa5e81959d34e7962e982cb16079c4dcf256312076200acc66d7fc83dca"}
Sep 10 04:47:50.376707 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:50.376655    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b5848dcdb-f95p8" podStartSLOduration=0.840902677 podStartE2EDuration="3.376641656s" totalImagesPullingTime="2.535738979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:47:50.375625277 +0000 UTC m=+407.137394458" watchObservedRunningTime="2026-09-10 04:47:50.376641656 +0000 UTC m=+407.138410842"
Sep 10 04:47:53.497695 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:53.497653    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:53.602256 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:53.602215    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:54.093021 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:54.092982    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:54.190680 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:54.190650    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:54.512872 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:54.512836    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:57.916387 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:57.916352    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jxfkk"]
Sep 10 04:47:57.919645 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:57.919629    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-jxfkk"
Sep 10 04:47:57.922229 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:57.922211    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 10 04:47:57.922291 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:57.922236    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-t748m\""
Sep 10 04:47:57.927942 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:57.927916    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jxfkk"]
Sep 10 04:47:57.986452 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:57.986416    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:58.059643 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.059606    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/822d9fab-f302-4685-93e1-ef41d09cfd1a-cert\") pod \"odh-model-controller-7998c67dc4-jxfkk\" (UID: \"822d9fab-f302-4685-93e1-ef41d09cfd1a\") " pod="opendatahub/odh-model-controller-7998c67dc4-jxfkk"
Sep 10 04:47:58.059806 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.059698    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77rrp\" (UniqueName: \"kubernetes.io/projected/822d9fab-f302-4685-93e1-ef41d09cfd1a-kube-api-access-77rrp\") pod \"odh-model-controller-7998c67dc4-jxfkk\" (UID: \"822d9fab-f302-4685-93e1-ef41d09cfd1a\") " pod="opendatahub/odh-model-controller-7998c67dc4-jxfkk"
Sep 10 04:47:58.086555 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.086519    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:58.160767 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.160736    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/822d9fab-f302-4685-93e1-ef41d09cfd1a-cert\") pod \"odh-model-controller-7998c67dc4-jxfkk\" (UID: \"822d9fab-f302-4685-93e1-ef41d09cfd1a\") " pod="opendatahub/odh-model-controller-7998c67dc4-jxfkk"
Sep 10 04:47:58.160953 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.160824    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77rrp\" (UniqueName: \"kubernetes.io/projected/822d9fab-f302-4685-93e1-ef41d09cfd1a-kube-api-access-77rrp\") pod \"odh-model-controller-7998c67dc4-jxfkk\" (UID: \"822d9fab-f302-4685-93e1-ef41d09cfd1a\") " pod="opendatahub/odh-model-controller-7998c67dc4-jxfkk"
Sep 10 04:47:58.163731 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.163702    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/822d9fab-f302-4685-93e1-ef41d09cfd1a-cert\") pod \"odh-model-controller-7998c67dc4-jxfkk\" (UID: \"822d9fab-f302-4685-93e1-ef41d09cfd1a\") " pod="opendatahub/odh-model-controller-7998c67dc4-jxfkk"
Sep 10 04:47:58.169202 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.169134    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77rrp\" (UniqueName: \"kubernetes.io/projected/822d9fab-f302-4685-93e1-ef41d09cfd1a-kube-api-access-77rrp\") pod \"odh-model-controller-7998c67dc4-jxfkk\" (UID: \"822d9fab-f302-4685-93e1-ef41d09cfd1a\") " pod="opendatahub/odh-model-controller-7998c67dc4-jxfkk"
Sep 10 04:47:58.187502 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.187472    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:58.230021 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.229995    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-jxfkk"
Sep 10 04:47:58.287485 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.287388    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:58.501275 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.501104    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6r8nx"]
Sep 10 04:47:58.503856 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:47:58.503827    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822d9fab_f302_4685_93e1_ef41d09cfd1a.slice/crio-83e457616245968cf3be88db14d4570c2b72aa8e4f3c8b45d43901c71bd8bb41 WatchSource:0}: Error finding container 83e457616245968cf3be88db14d4570c2b72aa8e4f3c8b45d43901c71bd8bb41: Status 404 returned error can't find the container with id 83e457616245968cf3be88db14d4570c2b72aa8e4f3c8b45d43901c71bd8bb41
Sep 10 04:47:58.505500 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.505480    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jxfkk"]
Sep 10 04:47:58.505586 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.505568    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-6r8nx"
Sep 10 04:47:58.508096 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.508078    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 10 04:47:58.515917 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.515889    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6r8nx"]
Sep 10 04:47:58.519221 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.519138    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:58.665744 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.665705    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bfb57ba-1de6-45ef-9e88-28d442ad2131-tls-certs\") pod \"model-serving-api-54d89bd5fc-6r8nx\" (UID: \"2bfb57ba-1de6-45ef-9e88-28d442ad2131\") " pod="opendatahub/model-serving-api-54d89bd5fc-6r8nx"
Sep 10 04:47:58.665744 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.665745    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpdnm\" (UniqueName: \"kubernetes.io/projected/2bfb57ba-1de6-45ef-9e88-28d442ad2131-kube-api-access-hpdnm\") pod \"model-serving-api-54d89bd5fc-6r8nx\" (UID: \"2bfb57ba-1de6-45ef-9e88-28d442ad2131\") " pod="opendatahub/model-serving-api-54d89bd5fc-6r8nx"
Sep 10 04:47:58.686388 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.686354    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:47:58.766652 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.766562    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bfb57ba-1de6-45ef-9e88-28d442ad2131-tls-certs\") pod \"model-serving-api-54d89bd5fc-6r8nx\" (UID: \"2bfb57ba-1de6-45ef-9e88-28d442ad2131\") " pod="opendatahub/model-serving-api-54d89bd5fc-6r8nx"
Sep 10 04:47:58.766652 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.766604    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpdnm\" (UniqueName: \"kubernetes.io/projected/2bfb57ba-1de6-45ef-9e88-28d442ad2131-kube-api-access-hpdnm\") pod \"model-serving-api-54d89bd5fc-6r8nx\" (UID: \"2bfb57ba-1de6-45ef-9e88-28d442ad2131\") " pod="opendatahub/model-serving-api-54d89bd5fc-6r8nx"
Sep 10 04:47:58.769265 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.769240    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bfb57ba-1de6-45ef-9e88-28d442ad2131-tls-certs\") pod \"model-serving-api-54d89bd5fc-6r8nx\" (UID: \"2bfb57ba-1de6-45ef-9e88-28d442ad2131\") " pod="opendatahub/model-serving-api-54d89bd5fc-6r8nx"
Sep 10 04:47:58.774481 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.774452    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpdnm\" (UniqueName: \"kubernetes.io/projected/2bfb57ba-1de6-45ef-9e88-28d442ad2131-kube-api-access-hpdnm\") pod \"model-serving-api-54d89bd5fc-6r8nx\" (UID: \"2bfb57ba-1de6-45ef-9e88-28d442ad2131\") " pod="opendatahub/model-serving-api-54d89bd5fc-6r8nx"
Sep 10 04:47:58.821112 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:58.821041    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-6r8nx"
Sep 10 04:47:59.092395 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:59.092349    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6r8nx"]
Sep 10 04:47:59.095122 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:47:59.095089    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfb57ba_1de6_45ef_9e88_28d442ad2131.slice/crio-69581468b146e5ddbf592c84d4cd952ee4285f43301be8edf2ebd2eab4320d74 WatchSource:0}: Error finding container 69581468b146e5ddbf592c84d4cd952ee4285f43301be8edf2ebd2eab4320d74: Status 404 returned error can't find the container with id 69581468b146e5ddbf592c84d4cd952ee4285f43301be8edf2ebd2eab4320d74
Sep 10 04:47:59.397156 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:59.397047    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-jxfkk" event={"ID":"822d9fab-f302-4685-93e1-ef41d09cfd1a","Type":"ContainerStarted","Data":"83e457616245968cf3be88db14d4570c2b72aa8e4f3c8b45d43901c71bd8bb41"}
Sep 10 04:47:59.398797 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:47:59.398766    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-6r8nx" event={"ID":"2bfb57ba-1de6-45ef-9e88-28d442ad2131","Type":"ContainerStarted","Data":"69581468b146e5ddbf592c84d4cd952ee4285f43301be8edf2ebd2eab4320d74"}
Sep 10 04:48:02.413139 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:02.413098    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-6r8nx" event={"ID":"2bfb57ba-1de6-45ef-9e88-28d442ad2131","Type":"ContainerStarted","Data":"09f1aff0e93ec98e87144b35743556888ef87861c2e8cd3fc0882b754bb760df"}
Sep 10 04:48:02.413553 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:02.413252    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-6r8nx"
Sep 10 04:48:02.414601 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:02.414579    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-jxfkk" event={"ID":"822d9fab-f302-4685-93e1-ef41d09cfd1a","Type":"ContainerStarted","Data":"93a07290dec8ba0825209d3520ea7b34a11b99efc40254b122bc67bd5b38ae1e"}
Sep 10 04:48:02.414730 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:02.414709    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-jxfkk"
Sep 10 04:48:02.430781 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:02.430731    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-6r8nx" podStartSLOduration=1.902205823 podStartE2EDuration="4.430717615s" totalImagesPullingTime="2.528511792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:48:02.429430831 +0000 UTC m=+419.191200010" watchObservedRunningTime="2026-09-10 04:48:02.430717615 +0000 UTC m=+419.192486788"
Sep 10 04:48:02.449012 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:02.445256    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-jxfkk" podStartSLOduration=2.322131946 podStartE2EDuration="5.445236936s" totalImagesPullingTime="3.12310499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:48:02.443157237 +0000 UTC m=+419.204926441" watchObservedRunningTime="2026-09-10 04:48:02.445236936 +0000 UTC m=+419.207006117"
Sep 10 04:48:05.673475 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:05.673437    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5fb77b59f6-pkxxf"]
Sep 10 04:48:05.678049 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:05.678030    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-pkxxf"
Sep 10 04:48:05.682137 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:05.682113    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hv8zz\""
Sep 10 04:48:05.682137 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:05.682129    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 04:48:05.682341 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:05.682151    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 04:48:05.682341 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:05.682165    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 04:48:05.682341 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:05.682207    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 04:48:05.682341 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:05.682243    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 04:48:05.685237 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:05.685132    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5fb77b59f6-pkxxf"]
Sep 10 04:48:05.829136 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:05.829089    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e92dba81-34ab-42c7-8601-f0a0332145b7-manager-config\") pod \"lws-controller-manager-5fb77b59f6-pkxxf\" (UID: \"e92dba81-34ab-42c7-8601-f0a0332145b7\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-pkxxf"
Sep 10 04:48:05.829301 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:05.829154    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wznrg\" (UniqueName: \"kubernetes.io/projected/e92dba81-34ab-42c7-8601-f0a0332145b7-kube-api-access-wznrg\") pod \"lws-controller-manager-5fb77b59f6-pkxxf\" (UID: \"e92dba81-34ab-42c7-8601-f0a0332145b7\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-pkxxf"
Sep 10 04:48:05.829301 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:05.829212    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e92dba81-34ab-42c7-8601-f0a0332145b7-metrics-cert\") pod \"lws-controller-manager-5fb77b59f6-pkxxf\" (UID: \"e92dba81-34ab-42c7-8601-f0a0332145b7\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-pkxxf"
Sep 10 04:48:05.829387 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:05.829283    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e92dba81-34ab-42c7-8601-f0a0332145b7-cert\") pod \"lws-controller-manager-5fb77b59f6-pkxxf\" (UID: \"e92dba81-34ab-42c7-8601-f0a0332145b7\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-pkxxf"
Sep 10 04:48:05.929996 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:05.929899    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e92dba81-34ab-42c7-8601-f0a0332145b7-manager-config\") pod \"lws-controller-manager-5fb77b59f6-pkxxf\" (UID: \"e92dba81-34ab-42c7-8601-f0a0332145b7\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-pkxxf"
Sep 10 04:48:05.929996 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:05.929963    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wznrg\" (UniqueName: \"kubernetes.io/projected/e92dba81-34ab-42c7-8601-f0a0332145b7-kube-api-access-wznrg\") pod \"lws-controller-manager-5fb77b59f6-pkxxf\" (UID: \"e92dba81-34ab-42c7-8601-f0a0332145b7\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-pkxxf"
Sep 10 04:48:05.930205 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:05.930014    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e92dba81-34ab-42c7-8601-f0a0332145b7-metrics-cert\") pod \"lws-controller-manager-5fb77b59f6-pkxxf\" (UID: \"e92dba81-34ab-42c7-8601-f0a0332145b7\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-pkxxf"
Sep 10 04:48:05.930205 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:05.930046    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e92dba81-34ab-42c7-8601-f0a0332145b7-cert\") pod \"lws-controller-manager-5fb77b59f6-pkxxf\" (UID: \"e92dba81-34ab-42c7-8601-f0a0332145b7\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-pkxxf"
Sep 10 04:48:05.930680 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:05.930646    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e92dba81-34ab-42c7-8601-f0a0332145b7-manager-config\") pod \"lws-controller-manager-5fb77b59f6-pkxxf\" (UID: \"e92dba81-34ab-42c7-8601-f0a0332145b7\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-pkxxf"
Sep 10 04:48:05.932596 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:05.932563    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e92dba81-34ab-42c7-8601-f0a0332145b7-metrics-cert\") pod \"lws-controller-manager-5fb77b59f6-pkxxf\" (UID: \"e92dba81-34ab-42c7-8601-f0a0332145b7\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-pkxxf"
Sep 10 04:48:05.932596 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:05.932585    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e92dba81-34ab-42c7-8601-f0a0332145b7-cert\") pod \"lws-controller-manager-5fb77b59f6-pkxxf\" (UID: \"e92dba81-34ab-42c7-8601-f0a0332145b7\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-pkxxf"
Sep 10 04:48:05.944859 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:05.944832    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wznrg\" (UniqueName: \"kubernetes.io/projected/e92dba81-34ab-42c7-8601-f0a0332145b7-kube-api-access-wznrg\") pod \"lws-controller-manager-5fb77b59f6-pkxxf\" (UID: \"e92dba81-34ab-42c7-8601-f0a0332145b7\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-pkxxf"
Sep 10 04:48:05.987782 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:05.987752    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-pkxxf"
Sep 10 04:48:06.258963 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:06.258939    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5fb77b59f6-pkxxf"]
Sep 10 04:48:06.261127 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:48:06.261100    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92dba81_34ab_42c7_8601_f0a0332145b7.slice/crio-79c9666b5729da206fa2f490aae283e05f5fcf902e483ba1fcd727c7d7200daf WatchSource:0}: Error finding container 79c9666b5729da206fa2f490aae283e05f5fcf902e483ba1fcd727c7d7200daf: Status 404 returned error can't find the container with id 79c9666b5729da206fa2f490aae283e05f5fcf902e483ba1fcd727c7d7200daf
Sep 10 04:48:06.430994 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:06.430956    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-pkxxf" event={"ID":"e92dba81-34ab-42c7-8601-f0a0332145b7","Type":"ContainerStarted","Data":"79c9666b5729da206fa2f490aae283e05f5fcf902e483ba1fcd727c7d7200daf"}
Sep 10 04:48:08.149465 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:08.149288    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:48:08.244716 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:08.244676    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:48:08.590996 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:08.590954    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:48:08.650134 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:08.650094    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:48:09.448295 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:09.448261    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-pkxxf" event={"ID":"e92dba81-34ab-42c7-8601-f0a0332145b7","Type":"ContainerStarted","Data":"c4eb8ca1bfe6b7f11ec9bf6891b938c004a5f09e9c5abf04cc3c93b8c4104c09"}
Sep 10 04:48:09.448659 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:09.448395    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-pkxxf"
Sep 10 04:48:09.468970 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:09.468910    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-pkxxf" podStartSLOduration=1.803626395 podStartE2EDuration="4.468892727s" totalImagesPullingTime="2.665266332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:48:09.467778568 +0000 UTC m=+426.229547753" watchObservedRunningTime="2026-09-10 04:48:09.468892727 +0000 UTC m=+426.230661908"
Sep 10 04:48:09.849496 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:09.849395    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:48:09.944551 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:09.944515    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:48:13.420667 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:13.420639    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-jxfkk"
Sep 10 04:48:13.422525 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:13.422505    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-6r8nx"
Sep 10 04:48:13.964799 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:13.964763    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:48:20.454508 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:20.454473    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-pkxxf"
Sep 10 04:48:34.037289 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:34.037207    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:48:39.201902 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:39.201861    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:48:39.264653 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:39.264617    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:48:41.056815 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:41.056780    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:48:59.123879 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:59.123837    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:48:59.218754 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:48:59.218712    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:13.671510 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:13.671474    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:14.480002 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:14.479968    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:17.705378 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:17.705341    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:17.918611 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:17.918574    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:19.355413 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:19.355373    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:19.885202 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:19.885169    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:20.331122 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:20.331081    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:20.677345 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:20.677262    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:21.033917 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:21.033879    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:21.139833 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:21.139800    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:24.146755 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:24.146710    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:25.563181 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:25.563146    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:27.131132 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:27.131086    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:27.227586 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:27.227552    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:27.492929 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:27.492895    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:27.593621 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:27.593592    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:30.754451 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:30.754411    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:31.013885 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:31.013771    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:31.116993 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:31.116951    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:31.214328 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:31.214289    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:36.621556 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:36.621512    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:36.721683 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:36.721643    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:36.785662 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:36.785619    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:45.223944 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:45.223902    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:45.330514 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:45.330476    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:49.408608 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:49.408574    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:49.458142 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:49.458110    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:50.013527 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:50.013493    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:50.066758 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:50.066728    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:53.836082 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:53.836022    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:53.933363 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:53.933332    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:59.509116 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:59.509010    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:49:59.568016 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:49:59.567975    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:00.016052 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:00.016009    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:00.116671 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:00.116640    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:00.613488 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:00.613453    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:00.714662 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:00.714627    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:08.015887 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:08.015841    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:08.116573 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:08.116542    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:09.746150 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:09.746113    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:09.974840 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:09.974806    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:10.227351 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:10.227311    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:10.317722 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:10.317687    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:23.685590 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:23.685552    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:23.798245 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:23.798211    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:23.851560 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:23.851530    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:24.476602 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:24.476568    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8dljf"]
Sep 10 04:50:24.478674 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:24.478646    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-8dljf"
Sep 10 04:50:24.481280 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:24.481260    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 04:50:24.481397 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:24.481262    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-phhzd\""
Sep 10 04:50:24.482255 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:24.482238    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 04:50:24.486279 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:24.486255    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8dljf"]
Sep 10 04:50:24.519047 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:24.519015    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:24.545103 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:24.545058    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k56xq\" (UniqueName: \"kubernetes.io/projected/afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87-kube-api-access-k56xq\") pod \"authorino-7bcfd8f9cd-8dljf\" (UID: \"afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87\") " pod="kuadrant-system/authorino-7bcfd8f9cd-8dljf"
Sep 10 04:50:24.585620 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:24.585591    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:24.595426 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:24.595401    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-d9gsl"]
Sep 10 04:50:24.597420 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:24.597406    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-d9gsl"
Sep 10 04:50:24.606798 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:24.606777    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-d9gsl"]
Sep 10 04:50:24.645873 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:24.645842    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k56xq\" (UniqueName: \"kubernetes.io/projected/afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87-kube-api-access-k56xq\") pod \"authorino-7bcfd8f9cd-8dljf\" (UID: \"afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87\") " pod="kuadrant-system/authorino-7bcfd8f9cd-8dljf"
Sep 10 04:50:24.645976 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:24.645934    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ncm4\" (UniqueName: \"kubernetes.io/projected/566ca51e-c2b1-4391-b8a8-fa761b7fe77f-kube-api-access-7ncm4\") pod \"authorino-59c974bd48-d9gsl\" (UID: \"566ca51e-c2b1-4391-b8a8-fa761b7fe77f\") " pod="kuadrant-system/authorino-59c974bd48-d9gsl"
Sep 10 04:50:24.653168 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:24.653142    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k56xq\" (UniqueName: \"kubernetes.io/projected/afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87-kube-api-access-k56xq\") pod \"authorino-7bcfd8f9cd-8dljf\" (UID: \"afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87\") " pod="kuadrant-system/authorino-7bcfd8f9cd-8dljf"
Sep 10 04:50:24.747262 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:24.747176    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ncm4\" (UniqueName: \"kubernetes.io/projected/566ca51e-c2b1-4391-b8a8-fa761b7fe77f-kube-api-access-7ncm4\") pod \"authorino-59c974bd48-d9gsl\" (UID: \"566ca51e-c2b1-4391-b8a8-fa761b7fe77f\") " pod="kuadrant-system/authorino-59c974bd48-d9gsl"
Sep 10 04:50:24.755277 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:24.755254    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ncm4\" (UniqueName: \"kubernetes.io/projected/566ca51e-c2b1-4391-b8a8-fa761b7fe77f-kube-api-access-7ncm4\") pod \"authorino-59c974bd48-d9gsl\" (UID: \"566ca51e-c2b1-4391-b8a8-fa761b7fe77f\") " pod="kuadrant-system/authorino-59c974bd48-d9gsl"
Sep 10 04:50:24.788137 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:24.788110    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-8dljf"
Sep 10 04:50:24.905855 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:24.905819    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-d9gsl"
Sep 10 04:50:25.053439 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:25.053368    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8dljf"]
Sep 10 04:50:25.056256 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:50:25.056230    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafb7c7cc_4d49_4fc5_a6ec_03e4fdfa5d87.slice/crio-00dd0b0f2f39b874fa6996a4c2b554ee7638d7006954af4937959d3d37547cc2 WatchSource:0}: Error finding container 00dd0b0f2f39b874fa6996a4c2b554ee7638d7006954af4937959d3d37547cc2: Status 404 returned error can't find the container with id 00dd0b0f2f39b874fa6996a4c2b554ee7638d7006954af4937959d3d37547cc2
Sep 10 04:50:25.172406 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:25.172384    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-d9gsl"]
Sep 10 04:50:25.174139 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:50:25.174109    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod566ca51e_c2b1_4391_b8a8_fa761b7fe77f.slice/crio-9e898bf85867f1be3f9547c924de66ebc924c045ad85ecdd637fa2f258bf6162 WatchSource:0}: Error finding container 9e898bf85867f1be3f9547c924de66ebc924c045ad85ecdd637fa2f258bf6162: Status 404 returned error can't find the container with id 9e898bf85867f1be3f9547c924de66ebc924c045ad85ecdd637fa2f258bf6162
Sep 10 04:50:25.986743 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:25.986706    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-d9gsl" event={"ID":"566ca51e-c2b1-4391-b8a8-fa761b7fe77f","Type":"ContainerStarted","Data":"9e898bf85867f1be3f9547c924de66ebc924c045ad85ecdd637fa2f258bf6162"}
Sep 10 04:50:25.988528 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:25.988487    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-8dljf" event={"ID":"afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87","Type":"ContainerStarted","Data":"00dd0b0f2f39b874fa6996a4c2b554ee7638d7006954af4937959d3d37547cc2"}
Sep 10 04:50:28.011841 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:28.011800    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-d9gsl" event={"ID":"566ca51e-c2b1-4391-b8a8-fa761b7fe77f","Type":"ContainerStarted","Data":"0087c741447c84431d862145b00cec071e2009dabf4aebb8361864f45a8b459c"}
Sep 10 04:50:28.013899 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:28.013871    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-8dljf" event={"ID":"afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87","Type":"ContainerStarted","Data":"f51a79665db67adfea7796b75a99aefb828a731dc419e55b1f1fe0c5aa610f29"}
Sep 10 04:50:28.028278 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:28.028228    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-d9gsl" podStartSLOduration=1.3168545950000001 podStartE2EDuration="4.028210336s" totalImagesPullingTime="2.711355741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:50:28.027350156 +0000 UTC m=+564.789119465" watchObservedRunningTime="2026-09-10 04:50:28.028210336 +0000 UTC m=+564.789979519"
Sep 10 04:50:28.048212 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:28.048012    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-8dljf" podStartSLOduration=1.228844959 podStartE2EDuration="4.047990522s" totalImagesPullingTime="2.819145563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:50:28.046357749 +0000 UTC m=+564.808126924" watchObservedRunningTime="2026-09-10 04:50:28.047990522 +0000 UTC m=+564.809759706"
Sep 10 04:50:28.070577 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:28.070537    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8dljf"]
Sep 10 04:50:30.021796 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:30.021754    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-8dljf" podUID="afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87" containerName="authorino" containerID="cri-o://f51a79665db67adfea7796b75a99aefb828a731dc419e55b1f1fe0c5aa610f29" gracePeriod=30
Sep 10 04:50:30.263121 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:30.263096    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-8dljf"
Sep 10 04:50:30.285679 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:30.285617    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k56xq\" (UniqueName: \"kubernetes.io/projected/afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87-kube-api-access-k56xq\") pod \"afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87\" (UID: \"afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87\") "
Sep 10 04:50:30.287683 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:30.287659    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87-kube-api-access-k56xq" (OuterVolumeSpecName: "kube-api-access-k56xq") pod "afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87" (UID: "afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87"). InnerVolumeSpecName "kube-api-access-k56xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 04:50:30.386109 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:30.386060    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k56xq\" (UniqueName: \"kubernetes.io/projected/afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87-kube-api-access-k56xq\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:50:31.026607 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:31.026570    2575 generic.go:356] "Generic (PLEG): container finished" podID="afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87" containerID="f51a79665db67adfea7796b75a99aefb828a731dc419e55b1f1fe0c5aa610f29" exitCode=0
Sep 10 04:50:31.026607 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:31.026611    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-8dljf"
Sep 10 04:50:31.027033 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:31.026644    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-8dljf" event={"ID":"afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87","Type":"ContainerDied","Data":"f51a79665db67adfea7796b75a99aefb828a731dc419e55b1f1fe0c5aa610f29"}
Sep 10 04:50:31.027033 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:31.026690    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-8dljf" event={"ID":"afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87","Type":"ContainerDied","Data":"00dd0b0f2f39b874fa6996a4c2b554ee7638d7006954af4937959d3d37547cc2"}
Sep 10 04:50:31.027033 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:31.026710    2575 scope.go:117] "RemoveContainer" containerID="f51a79665db67adfea7796b75a99aefb828a731dc419e55b1f1fe0c5aa610f29"
Sep 10 04:50:31.034524 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:31.034333    2575 scope.go:117] "RemoveContainer" containerID="f51a79665db67adfea7796b75a99aefb828a731dc419e55b1f1fe0c5aa610f29"
Sep 10 04:50:31.034821 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:50:31.034795    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f51a79665db67adfea7796b75a99aefb828a731dc419e55b1f1fe0c5aa610f29\": container with ID starting with f51a79665db67adfea7796b75a99aefb828a731dc419e55b1f1fe0c5aa610f29 not found: ID does not exist" containerID="f51a79665db67adfea7796b75a99aefb828a731dc419e55b1f1fe0c5aa610f29"
Sep 10 04:50:31.034912 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:31.034835    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f51a79665db67adfea7796b75a99aefb828a731dc419e55b1f1fe0c5aa610f29"} err="failed to get container status \"f51a79665db67adfea7796b75a99aefb828a731dc419e55b1f1fe0c5aa610f29\": rpc error: code = NotFound desc = could not find container \"f51a79665db67adfea7796b75a99aefb828a731dc419e55b1f1fe0c5aa610f29\": container with ID starting with f51a79665db67adfea7796b75a99aefb828a731dc419e55b1f1fe0c5aa610f29 not found: ID does not exist"
Sep 10 04:50:31.045680 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:31.045659    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8dljf"]
Sep 10 04:50:31.047503 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:31.047482    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8dljf"]
Sep 10 04:50:31.861597 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:31.861531    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87" path="/var/lib/kubelet/pods/afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87/volumes"
Sep 10 04:50:37.440725 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:37.440689    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:43.481682 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:43.481640    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:43.581227 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:43.581180    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:44.136533 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:44.136495    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:46.363128 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:46.363050    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:53.977404 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:53.977360    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:54.106351 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.106311    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:54.216743 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.216707    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-5cwfc"]
Sep 10 04:50:54.217003 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.216984    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87" containerName="authorino"
Sep 10 04:50:54.217053 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.217005    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87" containerName="authorino"
Sep 10 04:50:54.217111 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.217075    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="afb7c7cc-4d49-4fc5-a6ec-03e4fdfa5d87" containerName="authorino"
Sep 10 04:50:54.225115 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.225087    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5cwfc"
Sep 10 04:50:54.225907 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.225880    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-5cwfc"]
Sep 10 04:50:54.263031 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.263006    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtpkb\" (UniqueName: \"kubernetes.io/projected/233dd0d9-43f4-4897-9f10-8bcfe8dfff4b-kube-api-access-dtpkb\") pod \"authorino-557f876f69-5cwfc\" (UID: \"233dd0d9-43f4-4897-9f10-8bcfe8dfff4b\") " pod="kuadrant-system/authorino-557f876f69-5cwfc"
Sep 10 04:50:54.283742 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.283714    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-5cwfc"]
Sep 10 04:50:54.283942 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:50:54.283925    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dtpkb], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-5cwfc" podUID="233dd0d9-43f4-4897-9f10-8bcfe8dfff4b"
Sep 10 04:50:54.310647 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.310619    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4g4qx"]
Sep 10 04:50:54.314192 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.314172    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-4g4qx"
Sep 10 04:50:54.316476 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.316458    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 04:50:54.322764 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.322740    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4g4qx"]
Sep 10 04:50:54.363600 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.363562    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e67b060a-5a45-4af4-b663-6c3529786c5c-tls-cert\") pod \"authorino-6d4dfc649f-4g4qx\" (UID: \"e67b060a-5a45-4af4-b663-6c3529786c5c\") " pod="kuadrant-system/authorino-6d4dfc649f-4g4qx"
Sep 10 04:50:54.363781 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.363617    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bbv2\" (UniqueName: \"kubernetes.io/projected/e67b060a-5a45-4af4-b663-6c3529786c5c-kube-api-access-4bbv2\") pod \"authorino-6d4dfc649f-4g4qx\" (UID: \"e67b060a-5a45-4af4-b663-6c3529786c5c\") " pod="kuadrant-system/authorino-6d4dfc649f-4g4qx"
Sep 10 04:50:54.363781 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.363663    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtpkb\" (UniqueName: \"kubernetes.io/projected/233dd0d9-43f4-4897-9f10-8bcfe8dfff4b-kube-api-access-dtpkb\") pod \"authorino-557f876f69-5cwfc\" (UID: \"233dd0d9-43f4-4897-9f10-8bcfe8dfff4b\") " pod="kuadrant-system/authorino-557f876f69-5cwfc"
Sep 10 04:50:54.374493 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.374462    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtpkb\" (UniqueName: \"kubernetes.io/projected/233dd0d9-43f4-4897-9f10-8bcfe8dfff4b-kube-api-access-dtpkb\") pod \"authorino-557f876f69-5cwfc\" (UID: \"233dd0d9-43f4-4897-9f10-8bcfe8dfff4b\") " pod="kuadrant-system/authorino-557f876f69-5cwfc"
Sep 10 04:50:54.464441 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.464411    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e67b060a-5a45-4af4-b663-6c3529786c5c-tls-cert\") pod \"authorino-6d4dfc649f-4g4qx\" (UID: \"e67b060a-5a45-4af4-b663-6c3529786c5c\") " pod="kuadrant-system/authorino-6d4dfc649f-4g4qx"
Sep 10 04:50:54.464567 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.464451    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bbv2\" (UniqueName: \"kubernetes.io/projected/e67b060a-5a45-4af4-b663-6c3529786c5c-kube-api-access-4bbv2\") pod \"authorino-6d4dfc649f-4g4qx\" (UID: \"e67b060a-5a45-4af4-b663-6c3529786c5c\") " pod="kuadrant-system/authorino-6d4dfc649f-4g4qx"
Sep 10 04:50:54.466842 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.466811    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e67b060a-5a45-4af4-b663-6c3529786c5c-tls-cert\") pod \"authorino-6d4dfc649f-4g4qx\" (UID: \"e67b060a-5a45-4af4-b663-6c3529786c5c\") " pod="kuadrant-system/authorino-6d4dfc649f-4g4qx"
Sep 10 04:50:54.472526 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.472501    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bbv2\" (UniqueName: \"kubernetes.io/projected/e67b060a-5a45-4af4-b663-6c3529786c5c-kube-api-access-4bbv2\") pod \"authorino-6d4dfc649f-4g4qx\" (UID: \"e67b060a-5a45-4af4-b663-6c3529786c5c\") " pod="kuadrant-system/authorino-6d4dfc649f-4g4qx"
Sep 10 04:50:54.483515 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.483462    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4g4qx"]
Sep 10 04:50:54.483683 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.483670    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-4g4qx"
Sep 10 04:50:54.511360 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.511332    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79576f484-dfgld"]
Sep 10 04:50:54.514928 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.514912    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79576f484-dfgld"
Sep 10 04:50:54.524251 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.524215    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79576f484-dfgld"]
Sep 10 04:50:54.565700 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.565671    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67150e9f-17e1-48d3-a27f-5fc4ae8c82f3-tls-cert\") pod \"authorino-79576f484-dfgld\" (UID: \"67150e9f-17e1-48d3-a27f-5fc4ae8c82f3\") " pod="kuadrant-system/authorino-79576f484-dfgld"
Sep 10 04:50:54.565834 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.565703    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mnf2\" (UniqueName: \"kubernetes.io/projected/67150e9f-17e1-48d3-a27f-5fc4ae8c82f3-kube-api-access-6mnf2\") pod \"authorino-79576f484-dfgld\" (UID: \"67150e9f-17e1-48d3-a27f-5fc4ae8c82f3\") " pod="kuadrant-system/authorino-79576f484-dfgld"
Sep 10 04:50:54.666184 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.666148    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67150e9f-17e1-48d3-a27f-5fc4ae8c82f3-tls-cert\") pod \"authorino-79576f484-dfgld\" (UID: \"67150e9f-17e1-48d3-a27f-5fc4ae8c82f3\") " pod="kuadrant-system/authorino-79576f484-dfgld"
Sep 10 04:50:54.666184 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.666185    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mnf2\" (UniqueName: \"kubernetes.io/projected/67150e9f-17e1-48d3-a27f-5fc4ae8c82f3-kube-api-access-6mnf2\") pod \"authorino-79576f484-dfgld\" (UID: \"67150e9f-17e1-48d3-a27f-5fc4ae8c82f3\") " pod="kuadrant-system/authorino-79576f484-dfgld"
Sep 10 04:50:54.668468 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.668449    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67150e9f-17e1-48d3-a27f-5fc4ae8c82f3-tls-cert\") pod \"authorino-79576f484-dfgld\" (UID: \"67150e9f-17e1-48d3-a27f-5fc4ae8c82f3\") " pod="kuadrant-system/authorino-79576f484-dfgld"
Sep 10 04:50:54.674083 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.674043    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mnf2\" (UniqueName: \"kubernetes.io/projected/67150e9f-17e1-48d3-a27f-5fc4ae8c82f3-kube-api-access-6mnf2\") pod \"authorino-79576f484-dfgld\" (UID: \"67150e9f-17e1-48d3-a27f-5fc4ae8c82f3\") " pod="kuadrant-system/authorino-79576f484-dfgld"
Sep 10 04:50:54.755676 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.755599    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4g4qx"]
Sep 10 04:50:54.758228 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:50:54.758202    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode67b060a_5a45_4af4_b663_6c3529786c5c.slice/crio-14f7d5fd5e383c5eb2a9b7610487388d558e0d64928cf76e8e726cbe324e581f WatchSource:0}: Error finding container 14f7d5fd5e383c5eb2a9b7610487388d558e0d64928cf76e8e726cbe324e581f: Status 404 returned error can't find the container with id 14f7d5fd5e383c5eb2a9b7610487388d558e0d64928cf76e8e726cbe324e581f
Sep 10 04:50:54.825233 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:54.825195    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79576f484-dfgld"
Sep 10 04:50:55.092147 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:55.092120    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79576f484-dfgld"]
Sep 10 04:50:55.094336 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:50:55.094305    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67150e9f_17e1_48d3_a27f_5fc4ae8c82f3.slice/crio-90dd1c04464baee76a5f91bfbf64a76143e828119e20e5d37a28e440b24e4d9f WatchSource:0}: Error finding container 90dd1c04464baee76a5f91bfbf64a76143e828119e20e5d37a28e440b24e4d9f: Status 404 returned error can't find the container with id 90dd1c04464baee76a5f91bfbf64a76143e828119e20e5d37a28e440b24e4d9f
Sep 10 04:50:55.119586 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:55.119554    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79576f484-dfgld" event={"ID":"67150e9f-17e1-48d3-a27f-5fc4ae8c82f3","Type":"ContainerStarted","Data":"90dd1c04464baee76a5f91bfbf64a76143e828119e20e5d37a28e440b24e4d9f"}
Sep 10 04:50:55.120920 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:55.120875    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5cwfc"
Sep 10 04:50:55.120920 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:55.120878    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-4g4qx" event={"ID":"e67b060a-5a45-4af4-b663-6c3529786c5c","Type":"ContainerStarted","Data":"14f7d5fd5e383c5eb2a9b7610487388d558e0d64928cf76e8e726cbe324e581f"}
Sep 10 04:50:55.127121 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:55.127098    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5cwfc"
Sep 10 04:50:55.170183 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:55.170160    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtpkb\" (UniqueName: \"kubernetes.io/projected/233dd0d9-43f4-4897-9f10-8bcfe8dfff4b-kube-api-access-dtpkb\") pod \"233dd0d9-43f4-4897-9f10-8bcfe8dfff4b\" (UID: \"233dd0d9-43f4-4897-9f10-8bcfe8dfff4b\") "
Sep 10 04:50:55.172047 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:55.172014    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233dd0d9-43f4-4897-9f10-8bcfe8dfff4b-kube-api-access-dtpkb" (OuterVolumeSpecName: "kube-api-access-dtpkb") pod "233dd0d9-43f4-4897-9f10-8bcfe8dfff4b" (UID: "233dd0d9-43f4-4897-9f10-8bcfe8dfff4b"). InnerVolumeSpecName "kube-api-access-dtpkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 04:50:55.271080 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:55.271025    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtpkb\" (UniqueName: \"kubernetes.io/projected/233dd0d9-43f4-4897-9f10-8bcfe8dfff4b-kube-api-access-dtpkb\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:50:56.126901 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:56.126866    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-4g4qx" event={"ID":"e67b060a-5a45-4af4-b663-6c3529786c5c","Type":"ContainerStarted","Data":"47c23ba0ac23f33c0bcb683bbc909ffc5b720684006f7bd25ceeaadbfc3da253"}
Sep 10 04:50:56.127322 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:56.126883    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6d4dfc649f-4g4qx" podUID="e67b060a-5a45-4af4-b663-6c3529786c5c" containerName="authorino" containerID="cri-o://47c23ba0ac23f33c0bcb683bbc909ffc5b720684006f7bd25ceeaadbfc3da253" gracePeriod=30
Sep 10 04:50:56.128510 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:56.128493    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5cwfc"
Sep 10 04:50:56.128620 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:56.128530    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79576f484-dfgld" event={"ID":"67150e9f-17e1-48d3-a27f-5fc4ae8c82f3","Type":"ContainerStarted","Data":"0228a62407c8943c06afee3e009da0732b5fb5972cd28c7f391eb67bce5a8dd6"}
Sep 10 04:50:56.142197 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:56.142160    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d4dfc649f-4g4qx" podStartSLOduration=1.788491682 podStartE2EDuration="2.142147246s" totalImagesPullingTime="353.655564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:50:56.14076503 +0000 UTC m=+592.902534212" watchObservedRunningTime="2026-09-10 04:50:56.142147246 +0000 UTC m=+592.903916444"
Sep 10 04:50:56.162309 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:56.162288    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-5cwfc"]
Sep 10 04:50:56.166495 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:56.166468    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-5cwfc"]
Sep 10 04:50:56.178818 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:56.178779    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-79576f484-dfgld" podStartSLOduration=1.610467489 podStartE2EDuration="2.178766499s" totalImagesPullingTime="568.29901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:50:56.178317466 +0000 UTC m=+592.940086662" watchObservedRunningTime="2026-09-10 04:50:56.178766499 +0000 UTC m=+592.940535680"
Sep 10 04:50:56.204850 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:56.204818    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-d9gsl"]
Sep 10 04:50:56.205080 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:56.205040    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-d9gsl" podUID="566ca51e-c2b1-4391-b8a8-fa761b7fe77f" containerName="authorino" containerID="cri-o://0087c741447c84431d862145b00cec071e2009dabf4aebb8361864f45a8b459c" gracePeriod=30
Sep 10 04:50:56.467714 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:56.467685    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-4g4qx"
Sep 10 04:50:56.470276 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:56.470251    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-d9gsl"
Sep 10 04:50:56.580578 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:56.580540    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bbv2\" (UniqueName: \"kubernetes.io/projected/e67b060a-5a45-4af4-b663-6c3529786c5c-kube-api-access-4bbv2\") pod \"e67b060a-5a45-4af4-b663-6c3529786c5c\" (UID: \"e67b060a-5a45-4af4-b663-6c3529786c5c\") "
Sep 10 04:50:56.580770 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:56.580614    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e67b060a-5a45-4af4-b663-6c3529786c5c-tls-cert\") pod \"e67b060a-5a45-4af4-b663-6c3529786c5c\" (UID: \"e67b060a-5a45-4af4-b663-6c3529786c5c\") "
Sep 10 04:50:56.580770 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:56.580638    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ncm4\" (UniqueName: \"kubernetes.io/projected/566ca51e-c2b1-4391-b8a8-fa761b7fe77f-kube-api-access-7ncm4\") pod \"566ca51e-c2b1-4391-b8a8-fa761b7fe77f\" (UID: \"566ca51e-c2b1-4391-b8a8-fa761b7fe77f\") "
Sep 10 04:50:56.582674 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:56.582646    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566ca51e-c2b1-4391-b8a8-fa761b7fe77f-kube-api-access-7ncm4" (OuterVolumeSpecName: "kube-api-access-7ncm4") pod "566ca51e-c2b1-4391-b8a8-fa761b7fe77f" (UID: "566ca51e-c2b1-4391-b8a8-fa761b7fe77f"). InnerVolumeSpecName "kube-api-access-7ncm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 04:50:56.582830 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:56.582729    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67b060a-5a45-4af4-b663-6c3529786c5c-kube-api-access-4bbv2" (OuterVolumeSpecName: "kube-api-access-4bbv2") pod "e67b060a-5a45-4af4-b663-6c3529786c5c" (UID: "e67b060a-5a45-4af4-b663-6c3529786c5c"). InnerVolumeSpecName "kube-api-access-4bbv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 04:50:56.589866 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:56.589838    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e67b060a-5a45-4af4-b663-6c3529786c5c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e67b060a-5a45-4af4-b663-6c3529786c5c" (UID: "e67b060a-5a45-4af4-b663-6c3529786c5c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 04:50:56.681608 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:56.681513    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ncm4\" (UniqueName: \"kubernetes.io/projected/566ca51e-c2b1-4391-b8a8-fa761b7fe77f-kube-api-access-7ncm4\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:50:56.681608 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:56.681559    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bbv2\" (UniqueName: \"kubernetes.io/projected/e67b060a-5a45-4af4-b663-6c3529786c5c-kube-api-access-4bbv2\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:50:56.681608 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:56.681569    2575 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e67b060a-5a45-4af4-b663-6c3529786c5c-tls-cert\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:50:57.134274 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:57.134239    2575 generic.go:356] "Generic (PLEG): container finished" podID="566ca51e-c2b1-4391-b8a8-fa761b7fe77f" containerID="0087c741447c84431d862145b00cec071e2009dabf4aebb8361864f45a8b459c" exitCode=0
Sep 10 04:50:57.134684 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:57.134285    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-d9gsl"
Sep 10 04:50:57.134684 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:57.134320    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-d9gsl" event={"ID":"566ca51e-c2b1-4391-b8a8-fa761b7fe77f","Type":"ContainerDied","Data":"0087c741447c84431d862145b00cec071e2009dabf4aebb8361864f45a8b459c"}
Sep 10 04:50:57.134684 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:57.134365    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-d9gsl" event={"ID":"566ca51e-c2b1-4391-b8a8-fa761b7fe77f","Type":"ContainerDied","Data":"9e898bf85867f1be3f9547c924de66ebc924c045ad85ecdd637fa2f258bf6162"}
Sep 10 04:50:57.134684 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:57.134390    2575 scope.go:117] "RemoveContainer" containerID="0087c741447c84431d862145b00cec071e2009dabf4aebb8361864f45a8b459c"
Sep 10 04:50:57.135743 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:57.135578    2575 generic.go:356] "Generic (PLEG): container finished" podID="e67b060a-5a45-4af4-b663-6c3529786c5c" containerID="47c23ba0ac23f33c0bcb683bbc909ffc5b720684006f7bd25ceeaadbfc3da253" exitCode=0
Sep 10 04:50:57.135743 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:57.135629    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-4g4qx"
Sep 10 04:50:57.135743 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:57.135658    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-4g4qx" event={"ID":"e67b060a-5a45-4af4-b663-6c3529786c5c","Type":"ContainerDied","Data":"47c23ba0ac23f33c0bcb683bbc909ffc5b720684006f7bd25ceeaadbfc3da253"}
Sep 10 04:50:57.135743 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:57.135692    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-4g4qx" event={"ID":"e67b060a-5a45-4af4-b663-6c3529786c5c","Type":"ContainerDied","Data":"14f7d5fd5e383c5eb2a9b7610487388d558e0d64928cf76e8e726cbe324e581f"}
Sep 10 04:50:57.143496 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:57.143479    2575 scope.go:117] "RemoveContainer" containerID="0087c741447c84431d862145b00cec071e2009dabf4aebb8361864f45a8b459c"
Sep 10 04:50:57.143810 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:50:57.143774    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0087c741447c84431d862145b00cec071e2009dabf4aebb8361864f45a8b459c\": container with ID starting with 0087c741447c84431d862145b00cec071e2009dabf4aebb8361864f45a8b459c not found: ID does not exist" containerID="0087c741447c84431d862145b00cec071e2009dabf4aebb8361864f45a8b459c"
Sep 10 04:50:57.143926 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:57.143810    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0087c741447c84431d862145b00cec071e2009dabf4aebb8361864f45a8b459c"} err="failed to get container status \"0087c741447c84431d862145b00cec071e2009dabf4aebb8361864f45a8b459c\": rpc error: code = NotFound desc = could not find container \"0087c741447c84431d862145b00cec071e2009dabf4aebb8361864f45a8b459c\": container with ID starting with 0087c741447c84431d862145b00cec071e2009dabf4aebb8361864f45a8b459c not found: ID does not exist"
Sep 10 04:50:57.143926 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:57.143833    2575 scope.go:117] "RemoveContainer" containerID="47c23ba0ac23f33c0bcb683bbc909ffc5b720684006f7bd25ceeaadbfc3da253"
Sep 10 04:50:57.151224 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:57.151209    2575 scope.go:117] "RemoveContainer" containerID="47c23ba0ac23f33c0bcb683bbc909ffc5b720684006f7bd25ceeaadbfc3da253"
Sep 10 04:50:57.151464 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:50:57.151441    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c23ba0ac23f33c0bcb683bbc909ffc5b720684006f7bd25ceeaadbfc3da253\": container with ID starting with 47c23ba0ac23f33c0bcb683bbc909ffc5b720684006f7bd25ceeaadbfc3da253 not found: ID does not exist" containerID="47c23ba0ac23f33c0bcb683bbc909ffc5b720684006f7bd25ceeaadbfc3da253"
Sep 10 04:50:57.151516 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:57.151473    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c23ba0ac23f33c0bcb683bbc909ffc5b720684006f7bd25ceeaadbfc3da253"} err="failed to get container status \"47c23ba0ac23f33c0bcb683bbc909ffc5b720684006f7bd25ceeaadbfc3da253\": rpc error: code = NotFound desc = could not find container \"47c23ba0ac23f33c0bcb683bbc909ffc5b720684006f7bd25ceeaadbfc3da253\": container with ID starting with 47c23ba0ac23f33c0bcb683bbc909ffc5b720684006f7bd25ceeaadbfc3da253 not found: ID does not exist"
Sep 10 04:50:57.157748 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:57.157727    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-d9gsl"]
Sep 10 04:50:57.164476 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:57.164451    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-d9gsl"]
Sep 10 04:50:57.174551 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:57.174529    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4g4qx"]
Sep 10 04:50:57.178134 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:57.178114    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4g4qx"]
Sep 10 04:50:57.862775 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:57.862737    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="233dd0d9-43f4-4897-9f10-8bcfe8dfff4b" path="/var/lib/kubelet/pods/233dd0d9-43f4-4897-9f10-8bcfe8dfff4b/volumes"
Sep 10 04:50:57.863002 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:57.862989    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="566ca51e-c2b1-4391-b8a8-fa761b7fe77f" path="/var/lib/kubelet/pods/566ca51e-c2b1-4391-b8a8-fa761b7fe77f/volumes"
Sep 10 04:50:57.863374 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:57.863359    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e67b060a-5a45-4af4-b663-6c3529786c5c" path="/var/lib/kubelet/pods/e67b060a-5a45-4af4-b663-6c3529786c5c/volumes"
Sep 10 04:50:58.086352 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:58.086310    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:50:58.152710 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:50:58.152611    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:51:01.607394 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:01.607352    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:51:01.702538 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:01.702506    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:51:01.756056 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:01.756014    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:51:02.030594 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:02.030559    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:51:03.680400 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:03.680371    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2hd2_9d909143-4c70-4fe1-900a-3cd520db32e5/ovn-acl-logging/0.log"
Sep 10 04:51:03.794287 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:03.794261    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2hd2_9d909143-4c70-4fe1-900a-3cd520db32e5/ovn-acl-logging/0.log"
Sep 10 04:51:05.548746 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:05.548705    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:51:05.645464 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:05.645425    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:51:33.443368 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:33.443284    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:51:37.322793 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:37.322742    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:51:37.419894 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:37.419858    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:51:38.781560 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:38.781523    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:51:38.888527 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:38.888493    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:51:38.982144 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:38.982102    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:51:39.102503 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:39.102394    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:51:39.304663 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:39.304624    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:51:39.401506 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:39.401408    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:51:39.410109 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:39.410080    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-r72v6"]
Sep 10 04:51:39.410369 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:39.410357    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="566ca51e-c2b1-4391-b8a8-fa761b7fe77f" containerName="authorino"
Sep 10 04:51:39.410410 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:39.410370    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="566ca51e-c2b1-4391-b8a8-fa761b7fe77f" containerName="authorino"
Sep 10 04:51:39.410410 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:39.410378    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e67b060a-5a45-4af4-b663-6c3529786c5c" containerName="authorino"
Sep 10 04:51:39.410410 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:39.410384    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e67b060a-5a45-4af4-b663-6c3529786c5c" containerName="authorino"
Sep 10 04:51:39.410506 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:39.410444    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e67b060a-5a45-4af4-b663-6c3529786c5c" containerName="authorino"
Sep 10 04:51:39.410506 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:39.410453    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="566ca51e-c2b1-4391-b8a8-fa761b7fe77f" containerName="authorino"
Sep 10 04:51:39.413155 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:39.413138    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r72v6"
Sep 10 04:51:39.415826 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:39.415801    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 04:51:39.415943 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:39.415891    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-b6stt\""
Sep 10 04:51:39.423561 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:39.423530    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f884d1-f689-4baa-b91a-53debfb0e56f-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-r72v6\" (UID: \"40f884d1-f689-4baa-b91a-53debfb0e56f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r72v6"
Sep 10 04:51:39.423685 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:39.423584    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z86zs\" (UniqueName: \"kubernetes.io/projected/40f884d1-f689-4baa-b91a-53debfb0e56f-kube-api-access-z86zs\") pod \"payload-pre-processing-59fbc47bf8-r72v6\" (UID: \"40f884d1-f689-4baa-b91a-53debfb0e56f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r72v6"
Sep 10 04:51:39.423899 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:39.423878    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-r72v6"]
Sep 10 04:51:39.524121 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:39.524086    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f884d1-f689-4baa-b91a-53debfb0e56f-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-r72v6\" (UID: \"40f884d1-f689-4baa-b91a-53debfb0e56f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r72v6"
Sep 10 04:51:39.524273 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:39.524134    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z86zs\" (UniqueName: \"kubernetes.io/projected/40f884d1-f689-4baa-b91a-53debfb0e56f-kube-api-access-z86zs\") pod \"payload-pre-processing-59fbc47bf8-r72v6\" (UID: \"40f884d1-f689-4baa-b91a-53debfb0e56f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r72v6"
Sep 10 04:51:39.524787 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:39.524767    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f884d1-f689-4baa-b91a-53debfb0e56f-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-r72v6\" (UID: \"40f884d1-f689-4baa-b91a-53debfb0e56f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r72v6"
Sep 10 04:51:39.532509 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:39.532483    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z86zs\" (UniqueName: \"kubernetes.io/projected/40f884d1-f689-4baa-b91a-53debfb0e56f-kube-api-access-z86zs\") pod \"payload-pre-processing-59fbc47bf8-r72v6\" (UID: \"40f884d1-f689-4baa-b91a-53debfb0e56f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r72v6"
Sep 10 04:51:39.723250 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:39.723206    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r72v6"
Sep 10 04:51:39.996326 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:39.996252    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-r72v6"]
Sep 10 04:51:39.999506 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:51:39.999479    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f884d1_f689_4baa_b91a_53debfb0e56f.slice/crio-d7485f1f0fbdee5495c1bf4ba41401bdae9f9baef5a3cd11bf578e066f7d2fcc WatchSource:0}: Error finding container d7485f1f0fbdee5495c1bf4ba41401bdae9f9baef5a3cd11bf578e066f7d2fcc: Status 404 returned error can't find the container with id d7485f1f0fbdee5495c1bf4ba41401bdae9f9baef5a3cd11bf578e066f7d2fcc
Sep 10 04:51:40.302914 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:40.302828    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r72v6" event={"ID":"40f884d1-f689-4baa-b91a-53debfb0e56f","Type":"ContainerStarted","Data":"d7485f1f0fbdee5495c1bf4ba41401bdae9f9baef5a3cd11bf578e066f7d2fcc"}
Sep 10 04:51:43.317819 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:43.317783    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r72v6" event={"ID":"40f884d1-f689-4baa-b91a-53debfb0e56f","Type":"ContainerStarted","Data":"b963c5e1223f52d48d719e8f289b969da3ea75f8402a12bfbb044cb0afec2562"}
Sep 10 04:51:43.318266 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:43.317888    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r72v6"
Sep 10 04:51:43.334468 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:43.334413    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r72v6" podStartSLOduration=2.017216714 podStartE2EDuration="4.334397577s" totalImagesPullingTime="2.317180863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:51:43.333049119 +0000 UTC m=+640.094818301" watchObservedRunningTime="2026-09-10 04:51:43.334397577 +0000 UTC m=+640.096166758"
Sep 10 04:51:50.245318 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:50.245273    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:51:50.342884 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:50.342845    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:51:54.324250 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:54.324222    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r72v6"
Sep 10 04:51:56.420591 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:51:56.420553    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:52:14.283835 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.283799    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"]
Sep 10 04:52:14.287216 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.287190    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:14.290783 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.290762    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 04:52:14.290783 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.290778    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 04:52:14.290953 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.290778    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bstcg\""
Sep 10 04:52:14.290953 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.290765    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 10 04:52:14.297726 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.297704    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"]
Sep 10 04:52:14.316615 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.316587    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:52:14.410587 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.410556    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ecd3422-b133-42ac-9f55-4e486a3d1159-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvp56\" (UID: \"7ecd3422-b133-42ac-9f55-4e486a3d1159\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:14.410587 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.410587    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ecd3422-b133-42ac-9f55-4e486a3d1159-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvp56\" (UID: \"7ecd3422-b133-42ac-9f55-4e486a3d1159\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:14.410798 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.410622    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ecd3422-b133-42ac-9f55-4e486a3d1159-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvp56\" (UID: \"7ecd3422-b133-42ac-9f55-4e486a3d1159\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:14.410798 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.410675    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85bdp\" (UniqueName: \"kubernetes.io/projected/7ecd3422-b133-42ac-9f55-4e486a3d1159-kube-api-access-85bdp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvp56\" (UID: \"7ecd3422-b133-42ac-9f55-4e486a3d1159\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:14.410798 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.410726    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ecd3422-b133-42ac-9f55-4e486a3d1159-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvp56\" (UID: \"7ecd3422-b133-42ac-9f55-4e486a3d1159\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:14.410798 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.410749    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ecd3422-b133-42ac-9f55-4e486a3d1159-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvp56\" (UID: \"7ecd3422-b133-42ac-9f55-4e486a3d1159\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:14.511616 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.511579    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ecd3422-b133-42ac-9f55-4e486a3d1159-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvp56\" (UID: \"7ecd3422-b133-42ac-9f55-4e486a3d1159\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:14.511809 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.511639    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ecd3422-b133-42ac-9f55-4e486a3d1159-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvp56\" (UID: \"7ecd3422-b133-42ac-9f55-4e486a3d1159\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:14.511809 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.511657    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ecd3422-b133-42ac-9f55-4e486a3d1159-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvp56\" (UID: \"7ecd3422-b133-42ac-9f55-4e486a3d1159\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:14.511809 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.511689    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ecd3422-b133-42ac-9f55-4e486a3d1159-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvp56\" (UID: \"7ecd3422-b133-42ac-9f55-4e486a3d1159\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:14.511809 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.511707    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85bdp\" (UniqueName: \"kubernetes.io/projected/7ecd3422-b133-42ac-9f55-4e486a3d1159-kube-api-access-85bdp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvp56\" (UID: \"7ecd3422-b133-42ac-9f55-4e486a3d1159\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:14.512015 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.511823    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ecd3422-b133-42ac-9f55-4e486a3d1159-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvp56\" (UID: \"7ecd3422-b133-42ac-9f55-4e486a3d1159\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:14.512015 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.511992    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ecd3422-b133-42ac-9f55-4e486a3d1159-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvp56\" (UID: \"7ecd3422-b133-42ac-9f55-4e486a3d1159\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:14.512161 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.512036    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ecd3422-b133-42ac-9f55-4e486a3d1159-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvp56\" (UID: \"7ecd3422-b133-42ac-9f55-4e486a3d1159\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:14.512161 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.512118    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ecd3422-b133-42ac-9f55-4e486a3d1159-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvp56\" (UID: \"7ecd3422-b133-42ac-9f55-4e486a3d1159\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:14.514015 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.513993    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ecd3422-b133-42ac-9f55-4e486a3d1159-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvp56\" (UID: \"7ecd3422-b133-42ac-9f55-4e486a3d1159\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:14.514304 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.514286    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ecd3422-b133-42ac-9f55-4e486a3d1159-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvp56\" (UID: \"7ecd3422-b133-42ac-9f55-4e486a3d1159\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:14.519618 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.519593    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85bdp\" (UniqueName: \"kubernetes.io/projected/7ecd3422-b133-42ac-9f55-4e486a3d1159-kube-api-access-85bdp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvp56\" (UID: \"7ecd3422-b133-42ac-9f55-4e486a3d1159\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:14.598746 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.598667    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:14.870808 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:14.870778    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"]
Sep 10 04:52:14.872788 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:52:14.872749    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecd3422_b133_42ac_9f55_4e486a3d1159.slice/crio-8ad6d569e2b33e7a666770e7cdf4ac93cb6a6c7367b857e490173e9dceb1ecd9 WatchSource:0}: Error finding container 8ad6d569e2b33e7a666770e7cdf4ac93cb6a6c7367b857e490173e9dceb1ecd9: Status 404 returned error can't find the container with id 8ad6d569e2b33e7a666770e7cdf4ac93cb6a6c7367b857e490173e9dceb1ecd9
Sep 10 04:52:15.444864 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:15.444789    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56" event={"ID":"7ecd3422-b133-42ac-9f55-4e486a3d1159","Type":"ContainerStarted","Data":"8ad6d569e2b33e7a666770e7cdf4ac93cb6a6c7367b857e490173e9dceb1ecd9"}
Sep 10 04:52:16.450217 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:16.450177    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56" event={"ID":"7ecd3422-b133-42ac-9f55-4e486a3d1159","Type":"ContainerStarted","Data":"a2d041954c7736aab001c6faab9d5ec77abb2c690b897c93ea43576957e65591"}
Sep 10 04:52:16.467026 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:16.466973    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56" podStartSLOduration=1.383446538 podStartE2EDuration="2.466960348s" totalImagesPullingTime="1.08351381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:52:16.465381581 +0000 UTC m=+673.227150784" watchObservedRunningTime="2026-09-10 04:52:16.466960348 +0000 UTC m=+673.228729530"
Sep 10 04:52:21.519593 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:21.519549    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:52:24.599471 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:24.599433    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:24.599859 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:24.599524    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:24.612112 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:24.612087    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:25.500804 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:25.500769    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvp56"
Sep 10 04:52:33.221531 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:33.221494    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:52:39.194022 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.193985    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"]
Sep 10 04:52:39.200580 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.200553    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:39.203243 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.203213    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 10 04:52:39.207094 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.207050    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"]
Sep 10 04:52:39.221700 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.221671    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63241770-1f42-4b25-a827-b07d4a69f72b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88\" (UID: \"63241770-1f42-4b25-a827-b07d4a69f72b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:39.221835 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.221724    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63241770-1f42-4b25-a827-b07d4a69f72b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88\" (UID: \"63241770-1f42-4b25-a827-b07d4a69f72b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:39.221835 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.221765    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63241770-1f42-4b25-a827-b07d4a69f72b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88\" (UID: \"63241770-1f42-4b25-a827-b07d4a69f72b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:39.221949 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.221859    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63241770-1f42-4b25-a827-b07d4a69f72b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88\" (UID: \"63241770-1f42-4b25-a827-b07d4a69f72b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:39.221949 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.221898    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63241770-1f42-4b25-a827-b07d4a69f72b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88\" (UID: \"63241770-1f42-4b25-a827-b07d4a69f72b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:39.221949 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.221941    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rcfn\" (UniqueName: \"kubernetes.io/projected/63241770-1f42-4b25-a827-b07d4a69f72b-kube-api-access-2rcfn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88\" (UID: \"63241770-1f42-4b25-a827-b07d4a69f72b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:39.223311 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.223282    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:52:39.322985 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.322942    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63241770-1f42-4b25-a827-b07d4a69f72b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88\" (UID: \"63241770-1f42-4b25-a827-b07d4a69f72b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:39.322985 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.322993    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63241770-1f42-4b25-a827-b07d4a69f72b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88\" (UID: \"63241770-1f42-4b25-a827-b07d4a69f72b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:39.323265 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.323109    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63241770-1f42-4b25-a827-b07d4a69f72b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88\" (UID: \"63241770-1f42-4b25-a827-b07d4a69f72b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:39.323265 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.323195    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63241770-1f42-4b25-a827-b07d4a69f72b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88\" (UID: \"63241770-1f42-4b25-a827-b07d4a69f72b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:39.323265 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.323239    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63241770-1f42-4b25-a827-b07d4a69f72b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88\" (UID: \"63241770-1f42-4b25-a827-b07d4a69f72b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:39.323420 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.323396    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rcfn\" (UniqueName: \"kubernetes.io/projected/63241770-1f42-4b25-a827-b07d4a69f72b-kube-api-access-2rcfn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88\" (UID: \"63241770-1f42-4b25-a827-b07d4a69f72b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:39.323420 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.323412    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63241770-1f42-4b25-a827-b07d4a69f72b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88\" (UID: \"63241770-1f42-4b25-a827-b07d4a69f72b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:39.323536 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.323509    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63241770-1f42-4b25-a827-b07d4a69f72b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88\" (UID: \"63241770-1f42-4b25-a827-b07d4a69f72b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:39.323585 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.323532    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63241770-1f42-4b25-a827-b07d4a69f72b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88\" (UID: \"63241770-1f42-4b25-a827-b07d4a69f72b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:39.325392 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.325357    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63241770-1f42-4b25-a827-b07d4a69f72b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88\" (UID: \"63241770-1f42-4b25-a827-b07d4a69f72b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:39.325526 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.325510    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63241770-1f42-4b25-a827-b07d4a69f72b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88\" (UID: \"63241770-1f42-4b25-a827-b07d4a69f72b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:39.331093 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.331050    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rcfn\" (UniqueName: \"kubernetes.io/projected/63241770-1f42-4b25-a827-b07d4a69f72b-kube-api-access-2rcfn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88\" (UID: \"63241770-1f42-4b25-a827-b07d4a69f72b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:39.512846 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.512811    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:39.782035 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.782009    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"]
Sep 10 04:52:39.784554 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:52:39.784528    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63241770_1f42_4b25_a827_b07d4a69f72b.slice/crio-86d2700cf722280566f235a6e1974866661f5864f960c5e1237e117789a19bc4 WatchSource:0}: Error finding container 86d2700cf722280566f235a6e1974866661f5864f960c5e1237e117789a19bc4: Status 404 returned error can't find the container with id 86d2700cf722280566f235a6e1974866661f5864f960c5e1237e117789a19bc4
Sep 10 04:52:39.785836 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:39.785819    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 04:52:40.552703 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:40.552670    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88" event={"ID":"63241770-1f42-4b25-a827-b07d4a69f72b","Type":"ContainerStarted","Data":"d5a894f2ff7904a780556aab49cef674a13cf4ba01fe3047d067cf16ba55a75d"}
Sep 10 04:52:40.553218 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:40.552723    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88" event={"ID":"63241770-1f42-4b25-a827-b07d4a69f72b","Type":"ContainerStarted","Data":"86d2700cf722280566f235a6e1974866661f5864f960c5e1237e117789a19bc4"}
Sep 10 04:52:40.571501 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:40.571451    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88" podStartSLOduration=1.385388227 podStartE2EDuration="1.571434171s" totalImagesPullingTime="186.045944ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:52:40.570345384 +0000 UTC m=+697.332114562" watchObservedRunningTime="2026-09-10 04:52:40.571434171 +0000 UTC m=+697.333203353"
Sep 10 04:52:42.474754 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.474718    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"]
Sep 10 04:52:42.478023 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.478008    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:42.480503 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.480470    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 10 04:52:42.487990 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.487968    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"]
Sep 10 04:52:42.554252 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.554211    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h49pk\" (UniqueName: \"kubernetes.io/projected/3fd2862e-d95a-42f5-9924-bbcbf30a31c6-kube-api-access-h49pk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fjdxn\" (UID: \"3fd2862e-d95a-42f5-9924-bbcbf30a31c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:42.554429 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.554310    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fd2862e-d95a-42f5-9924-bbcbf30a31c6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fjdxn\" (UID: \"3fd2862e-d95a-42f5-9924-bbcbf30a31c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:42.554429 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.554333    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fd2862e-d95a-42f5-9924-bbcbf30a31c6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fjdxn\" (UID: \"3fd2862e-d95a-42f5-9924-bbcbf30a31c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:42.554429 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.554369    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fd2862e-d95a-42f5-9924-bbcbf30a31c6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fjdxn\" (UID: \"3fd2862e-d95a-42f5-9924-bbcbf30a31c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:42.554429 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.554386    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fd2862e-d95a-42f5-9924-bbcbf30a31c6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fjdxn\" (UID: \"3fd2862e-d95a-42f5-9924-bbcbf30a31c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:42.554429 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.554410    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fd2862e-d95a-42f5-9924-bbcbf30a31c6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fjdxn\" (UID: \"3fd2862e-d95a-42f5-9924-bbcbf30a31c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:42.591021 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.590986    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:52:42.655365 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.655328    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fd2862e-d95a-42f5-9924-bbcbf30a31c6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fjdxn\" (UID: \"3fd2862e-d95a-42f5-9924-bbcbf30a31c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:42.655365 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.655367    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fd2862e-d95a-42f5-9924-bbcbf30a31c6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fjdxn\" (UID: \"3fd2862e-d95a-42f5-9924-bbcbf30a31c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:42.655569 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.655394    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h49pk\" (UniqueName: \"kubernetes.io/projected/3fd2862e-d95a-42f5-9924-bbcbf30a31c6-kube-api-access-h49pk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fjdxn\" (UID: \"3fd2862e-d95a-42f5-9924-bbcbf30a31c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:42.655569 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.655453    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fd2862e-d95a-42f5-9924-bbcbf30a31c6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fjdxn\" (UID: \"3fd2862e-d95a-42f5-9924-bbcbf30a31c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:42.655569 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.655475    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fd2862e-d95a-42f5-9924-bbcbf30a31c6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fjdxn\" (UID: \"3fd2862e-d95a-42f5-9924-bbcbf30a31c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:42.655569 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.655507    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fd2862e-d95a-42f5-9924-bbcbf30a31c6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fjdxn\" (UID: \"3fd2862e-d95a-42f5-9924-bbcbf30a31c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:42.655842 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.655815    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fd2862e-d95a-42f5-9924-bbcbf30a31c6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fjdxn\" (UID: \"3fd2862e-d95a-42f5-9924-bbcbf30a31c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:42.655887 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.655827    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fd2862e-d95a-42f5-9924-bbcbf30a31c6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fjdxn\" (UID: \"3fd2862e-d95a-42f5-9924-bbcbf30a31c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:42.655928 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.655883    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fd2862e-d95a-42f5-9924-bbcbf30a31c6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fjdxn\" (UID: \"3fd2862e-d95a-42f5-9924-bbcbf30a31c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:42.657757 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.657727    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fd2862e-d95a-42f5-9924-bbcbf30a31c6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fjdxn\" (UID: \"3fd2862e-d95a-42f5-9924-bbcbf30a31c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:42.658009 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.657990    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fd2862e-d95a-42f5-9924-bbcbf30a31c6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fjdxn\" (UID: \"3fd2862e-d95a-42f5-9924-bbcbf30a31c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:42.664769 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.664738    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h49pk\" (UniqueName: \"kubernetes.io/projected/3fd2862e-d95a-42f5-9924-bbcbf30a31c6-kube-api-access-h49pk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fjdxn\" (UID: \"3fd2862e-d95a-42f5-9924-bbcbf30a31c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:42.788432 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:42.788344    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:43.059904 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:43.059880    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"]
Sep 10 04:52:43.061773 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:52:43.061741    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd2862e_d95a_42f5_9924_bbcbf30a31c6.slice/crio-e869a651f27f636f12236fbc13a460346ee3083805edab4d94be95f814463556 WatchSource:0}: Error finding container e869a651f27f636f12236fbc13a460346ee3083805edab4d94be95f814463556: Status 404 returned error can't find the container with id e869a651f27f636f12236fbc13a460346ee3083805edab4d94be95f814463556
Sep 10 04:52:43.565921 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:43.565881    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn" event={"ID":"3fd2862e-d95a-42f5-9924-bbcbf30a31c6","Type":"ContainerStarted","Data":"3405ca697ca2194047d66a98e358c8b9bb682829507bd83072f3ad42d15cd65f"}
Sep 10 04:52:43.566322 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:43.565935    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn" event={"ID":"3fd2862e-d95a-42f5-9924-bbcbf30a31c6","Type":"ContainerStarted","Data":"e869a651f27f636f12236fbc13a460346ee3083805edab4d94be95f814463556"}
Sep 10 04:52:43.583422 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:43.583375    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn" podStartSLOduration=1.40537777 podStartE2EDuration="1.583358768s" totalImagesPullingTime="177.980998ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:52:43.582205024 +0000 UTC m=+700.343974209" watchObservedRunningTime="2026-09-10 04:52:43.583358768 +0000 UTC m=+700.345127950"
Sep 10 04:52:48.392855 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.392823    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"]
Sep 10 04:52:48.396628 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.396610    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:48.399177 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.399150    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 04:52:48.403210 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.403186    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"]
Sep 10 04:52:48.422622 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.422594    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:52:48.503442 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.503406    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fgml9\" (UID: \"dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:48.503442 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.503445    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fgml9\" (UID: \"dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:48.503641 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.503465    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fgml9\" (UID: \"dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:48.503641 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.503537    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fgml9\" (UID: \"dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:48.503641 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.503589    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8ft8\" (UniqueName: \"kubernetes.io/projected/dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1-kube-api-access-f8ft8\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fgml9\" (UID: \"dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:48.503641 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.503620    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fgml9\" (UID: \"dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:48.604232 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.604201    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fgml9\" (UID: \"dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:48.604232 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.604235    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fgml9\" (UID: \"dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:48.604433 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.604257    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fgml9\" (UID: \"dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:48.604433 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.604296    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fgml9\" (UID: \"dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:48.604433 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.604327    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8ft8\" (UniqueName: \"kubernetes.io/projected/dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1-kube-api-access-f8ft8\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fgml9\" (UID: \"dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:48.604433 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.604362    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fgml9\" (UID: \"dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:48.604751 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.604716    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fgml9\" (UID: \"dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:48.604884 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.604790    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fgml9\" (UID: \"dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:48.604884 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.604818    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fgml9\" (UID: \"dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:48.606915 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.606891    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fgml9\" (UID: \"dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:48.607141 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.607125    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fgml9\" (UID: \"dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:48.612099 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.612051    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8ft8\" (UniqueName: \"kubernetes.io/projected/dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1-kube-api-access-f8ft8\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fgml9\" (UID: \"dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:48.707400 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.707369    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:48.976693 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:48.976666    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"]
Sep 10 04:52:48.979339 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:52:48.979308    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc12d19c_9c81_4c4f_8ddb_06cf04abfcc1.slice/crio-74e73eaec6da318e3936b256a868cdb497f649819299b3eb935c2ec4e541c7d3 WatchSource:0}: Error finding container 74e73eaec6da318e3936b256a868cdb497f649819299b3eb935c2ec4e541c7d3: Status 404 returned error can't find the container with id 74e73eaec6da318e3936b256a868cdb497f649819299b3eb935c2ec4e541c7d3
Sep 10 04:52:49.513061 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:49.513027    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:49.513418 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:49.513120    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:49.525627 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:49.525600    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:49.592032 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:49.591997    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9" event={"ID":"dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1","Type":"ContainerStarted","Data":"7651cb729386e9519a9e746fb54000410bc84a01d4a0ef121e7c622a3c663768"}
Sep 10 04:52:49.592189 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:49.592041    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9" event={"ID":"dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1","Type":"ContainerStarted","Data":"74e73eaec6da318e3936b256a868cdb497f649819299b3eb935c2ec4e541c7d3"}
Sep 10 04:52:49.603721 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:49.603690    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88"
Sep 10 04:52:49.607904 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:49.607853    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9" podStartSLOduration=1.419635493 podStartE2EDuration="1.607838201s" totalImagesPullingTime="188.202708ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:52:49.60649116 +0000 UTC m=+706.368260381" watchObservedRunningTime="2026-09-10 04:52:49.607838201 +0000 UTC m=+706.369607388"
Sep 10 04:52:52.788621 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:52.788588    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:52.789118 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:52.788676    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:52.801255 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:52.801234    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:53.619058 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:53.619023    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fjdxn"
Sep 10 04:52:58.708349 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:58.708268    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:58.708349 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:58.708318    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:58.720816 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:58.720790    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:52:59.644771 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:52:59.644743    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fgml9"
Sep 10 04:53:35.395704 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:53:35.395660    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:01.488460 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:01.488415    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:26.580796 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:26.580707    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-78fc98c8bf-kpwwf"]
Sep 10 04:54:26.583873 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:26.583856    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78fc98c8bf-kpwwf"
Sep 10 04:54:26.603229 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:26.603200    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78fc98c8bf-kpwwf"]
Sep 10 04:54:26.649553 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:26.649516    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzmrf\" (UniqueName: \"kubernetes.io/projected/a336d026-9d0e-4ac6-81d8-08e992654b98-kube-api-access-mzmrf\") pod \"authorino-78fc98c8bf-kpwwf\" (UID: \"a336d026-9d0e-4ac6-81d8-08e992654b98\") " pod="kuadrant-system/authorino-78fc98c8bf-kpwwf"
Sep 10 04:54:26.649717 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:26.649564    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a336d026-9d0e-4ac6-81d8-08e992654b98-tls-cert\") pod \"authorino-78fc98c8bf-kpwwf\" (UID: \"a336d026-9d0e-4ac6-81d8-08e992654b98\") " pod="kuadrant-system/authorino-78fc98c8bf-kpwwf"
Sep 10 04:54:26.750696 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:26.750659    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzmrf\" (UniqueName: \"kubernetes.io/projected/a336d026-9d0e-4ac6-81d8-08e992654b98-kube-api-access-mzmrf\") pod \"authorino-78fc98c8bf-kpwwf\" (UID: \"a336d026-9d0e-4ac6-81d8-08e992654b98\") " pod="kuadrant-system/authorino-78fc98c8bf-kpwwf"
Sep 10 04:54:26.750858 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:26.750716    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a336d026-9d0e-4ac6-81d8-08e992654b98-tls-cert\") pod \"authorino-78fc98c8bf-kpwwf\" (UID: \"a336d026-9d0e-4ac6-81d8-08e992654b98\") " pod="kuadrant-system/authorino-78fc98c8bf-kpwwf"
Sep 10 04:54:26.753255 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:26.753231    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a336d026-9d0e-4ac6-81d8-08e992654b98-tls-cert\") pod \"authorino-78fc98c8bf-kpwwf\" (UID: \"a336d026-9d0e-4ac6-81d8-08e992654b98\") " pod="kuadrant-system/authorino-78fc98c8bf-kpwwf"
Sep 10 04:54:26.764653 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:26.764630    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzmrf\" (UniqueName: \"kubernetes.io/projected/a336d026-9d0e-4ac6-81d8-08e992654b98-kube-api-access-mzmrf\") pod \"authorino-78fc98c8bf-kpwwf\" (UID: \"a336d026-9d0e-4ac6-81d8-08e992654b98\") " pod="kuadrant-system/authorino-78fc98c8bf-kpwwf"
Sep 10 04:54:26.892585 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:26.892501    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78fc98c8bf-kpwwf"
Sep 10 04:54:27.159235 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:27.159163    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78fc98c8bf-kpwwf"]
Sep 10 04:54:27.162979 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:54:27.162952    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda336d026_9d0e_4ac6_81d8_08e992654b98.slice/crio-87215a9e45a6a380d55b74c181aff26fbc4151fe0880fec9d9f6ffcccb6a8c85 WatchSource:0}: Error finding container 87215a9e45a6a380d55b74c181aff26fbc4151fe0880fec9d9f6ffcccb6a8c85: Status 404 returned error can't find the container with id 87215a9e45a6a380d55b74c181aff26fbc4151fe0880fec9d9f6ffcccb6a8c85
Sep 10 04:54:27.996017 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:27.995990    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78fc98c8bf-kpwwf" event={"ID":"a336d026-9d0e-4ac6-81d8-08e992654b98","Type":"ContainerStarted","Data":"87215a9e45a6a380d55b74c181aff26fbc4151fe0880fec9d9f6ffcccb6a8c85"}
Sep 10 04:54:29.003983 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:29.003933    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78fc98c8bf-kpwwf" event={"ID":"a336d026-9d0e-4ac6-81d8-08e992654b98","Type":"ContainerStarted","Data":"a76ff6afb132976f5da0c960b75ddddc00c643b0f117a22eaa68d1985e1e3c08"}
Sep 10 04:54:29.046197 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:29.046140    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-78fc98c8bf-kpwwf" podStartSLOduration=2.265360963 podStartE2EDuration="3.046120397s" totalImagesPullingTime="780.759434ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:54:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:54:29.021504282 +0000 UTC m=+805.783273464" watchObservedRunningTime="2026-09-10 04:54:29.046120397 +0000 UTC m=+805.807889587"
Sep 10 04:54:29.046967 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:29.046947    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79576f484-dfgld"]
Sep 10 04:54:29.047159 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:29.047138    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-79576f484-dfgld" podUID="67150e9f-17e1-48d3-a27f-5fc4ae8c82f3" containerName="authorino" containerID="cri-o://0228a62407c8943c06afee3e009da0732b5fb5972cd28c7f391eb67bce5a8dd6" gracePeriod=30
Sep 10 04:54:29.304874 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:29.304839    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79576f484-dfgld"
Sep 10 04:54:29.372523 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:29.372488    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67150e9f-17e1-48d3-a27f-5fc4ae8c82f3-tls-cert\") pod \"67150e9f-17e1-48d3-a27f-5fc4ae8c82f3\" (UID: \"67150e9f-17e1-48d3-a27f-5fc4ae8c82f3\") "
Sep 10 04:54:29.372707 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:29.372532    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mnf2\" (UniqueName: \"kubernetes.io/projected/67150e9f-17e1-48d3-a27f-5fc4ae8c82f3-kube-api-access-6mnf2\") pod \"67150e9f-17e1-48d3-a27f-5fc4ae8c82f3\" (UID: \"67150e9f-17e1-48d3-a27f-5fc4ae8c82f3\") "
Sep 10 04:54:29.374610 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:29.374576    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67150e9f-17e1-48d3-a27f-5fc4ae8c82f3-kube-api-access-6mnf2" (OuterVolumeSpecName: "kube-api-access-6mnf2") pod "67150e9f-17e1-48d3-a27f-5fc4ae8c82f3" (UID: "67150e9f-17e1-48d3-a27f-5fc4ae8c82f3"). InnerVolumeSpecName "kube-api-access-6mnf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 04:54:29.383291 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:29.383261    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67150e9f-17e1-48d3-a27f-5fc4ae8c82f3-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "67150e9f-17e1-48d3-a27f-5fc4ae8c82f3" (UID: "67150e9f-17e1-48d3-a27f-5fc4ae8c82f3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 04:54:29.473670 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:29.473632    2575 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67150e9f-17e1-48d3-a27f-5fc4ae8c82f3-tls-cert\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:54:29.473670 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:29.473664    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mnf2\" (UniqueName: \"kubernetes.io/projected/67150e9f-17e1-48d3-a27f-5fc4ae8c82f3-kube-api-access-6mnf2\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:54:29.554655 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:29.554563    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:29.653311 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:29.653259    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:30.010236 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:30.010200    2575 generic.go:356] "Generic (PLEG): container finished" podID="67150e9f-17e1-48d3-a27f-5fc4ae8c82f3" containerID="0228a62407c8943c06afee3e009da0732b5fb5972cd28c7f391eb67bce5a8dd6" exitCode=0
Sep 10 04:54:30.010648 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:30.010245    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79576f484-dfgld"
Sep 10 04:54:30.010648 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:30.010286    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79576f484-dfgld" event={"ID":"67150e9f-17e1-48d3-a27f-5fc4ae8c82f3","Type":"ContainerDied","Data":"0228a62407c8943c06afee3e009da0732b5fb5972cd28c7f391eb67bce5a8dd6"}
Sep 10 04:54:30.010648 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:30.010333    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79576f484-dfgld" event={"ID":"67150e9f-17e1-48d3-a27f-5fc4ae8c82f3","Type":"ContainerDied","Data":"90dd1c04464baee76a5f91bfbf64a76143e828119e20e5d37a28e440b24e4d9f"}
Sep 10 04:54:30.010648 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:30.010355    2575 scope.go:117] "RemoveContainer" containerID="0228a62407c8943c06afee3e009da0732b5fb5972cd28c7f391eb67bce5a8dd6"
Sep 10 04:54:30.018498 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:30.018478    2575 scope.go:117] "RemoveContainer" containerID="0228a62407c8943c06afee3e009da0732b5fb5972cd28c7f391eb67bce5a8dd6"
Sep 10 04:54:30.018756 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:54:30.018737    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0228a62407c8943c06afee3e009da0732b5fb5972cd28c7f391eb67bce5a8dd6\": container with ID starting with 0228a62407c8943c06afee3e009da0732b5fb5972cd28c7f391eb67bce5a8dd6 not found: ID does not exist" containerID="0228a62407c8943c06afee3e009da0732b5fb5972cd28c7f391eb67bce5a8dd6"
Sep 10 04:54:30.018803 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:30.018765    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0228a62407c8943c06afee3e009da0732b5fb5972cd28c7f391eb67bce5a8dd6"} err="failed to get container status \"0228a62407c8943c06afee3e009da0732b5fb5972cd28c7f391eb67bce5a8dd6\": rpc error: code = NotFound desc = could not find container \"0228a62407c8943c06afee3e009da0732b5fb5972cd28c7f391eb67bce5a8dd6\": container with ID starting with 0228a62407c8943c06afee3e009da0732b5fb5972cd28c7f391eb67bce5a8dd6 not found: ID does not exist"
Sep 10 04:54:30.047910 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:30.047877    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79576f484-dfgld"]
Sep 10 04:54:30.061210 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:30.061183    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-79576f484-dfgld"]
Sep 10 04:54:30.759636 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:30.759578    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:30.858668 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:30.858631    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:31.863421 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:31.863387    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67150e9f-17e1-48d3-a27f-5fc4ae8c82f3" path="/var/lib/kubelet/pods/67150e9f-17e1-48d3-a27f-5fc4ae8c82f3/volumes"
Sep 10 04:54:47.465509 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:47.465472    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:47.570253 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:47.570220    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:47.690566 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:47.690526    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:47.854515 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:47.854440    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:47.926795 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:47.926752    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:47.989652 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:47.989600    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:48.165936 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:48.165851    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:48.364283 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:48.364243    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:48.576568 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:48.576528    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:49.077984 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:49.077948    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:49.180137 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:49.180090    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:49.582526 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:49.582481    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:51.190232 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:51.190193    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:51.281226 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:51.281152    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:51.381192 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:51.381159    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:52.371280 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:52.371242    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:52.473487 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:52.473447    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:52.570387 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:52.570355    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:52.860384 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:52.860354    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:52.960265 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:52.960229    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:53.049887 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:53.049839    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5"]
Sep 10 04:54:53.050200 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:53.050187    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67150e9f-17e1-48d3-a27f-5fc4ae8c82f3" containerName="authorino"
Sep 10 04:54:53.050249 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:53.050202    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="67150e9f-17e1-48d3-a27f-5fc4ae8c82f3" containerName="authorino"
Sep 10 04:54:53.050288 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:53.050259    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="67150e9f-17e1-48d3-a27f-5fc4ae8c82f3" containerName="authorino"
Sep 10 04:54:53.054254 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:53.054238    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5"
Sep 10 04:54:53.057878 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:53.057849    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-7f2cd3\""
Sep 10 04:54:53.057990 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:53.057944    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-7f2cd3-dockercfg-fcghx\""
Sep 10 04:54:53.068017 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:53.067995    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5"]
Sep 10 04:54:53.178637 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:53.178556    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d0590b-d31b-41e8-9e9c-333030a3a96d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5\" (UID: \"f6d0590b-d31b-41e8-9e9c-333030a3a96d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5"
Sep 10 04:54:53.178637 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:53.178608    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kl7n\" (UniqueName: \"kubernetes.io/projected/f6d0590b-d31b-41e8-9e9c-333030a3a96d-kube-api-access-2kl7n\") pod \"payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5\" (UID: \"f6d0590b-d31b-41e8-9e9c-333030a3a96d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5"
Sep 10 04:54:53.279218 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:53.279162    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d0590b-d31b-41e8-9e9c-333030a3a96d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5\" (UID: \"f6d0590b-d31b-41e8-9e9c-333030a3a96d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5"
Sep 10 04:54:53.279428 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:53.279242    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kl7n\" (UniqueName: \"kubernetes.io/projected/f6d0590b-d31b-41e8-9e9c-333030a3a96d-kube-api-access-2kl7n\") pod \"payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5\" (UID: \"f6d0590b-d31b-41e8-9e9c-333030a3a96d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5"
Sep 10 04:54:53.279934 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:53.279909    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d0590b-d31b-41e8-9e9c-333030a3a96d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5\" (UID: \"f6d0590b-d31b-41e8-9e9c-333030a3a96d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5"
Sep 10 04:54:53.308599 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:53.308571    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kl7n\" (UniqueName: \"kubernetes.io/projected/f6d0590b-d31b-41e8-9e9c-333030a3a96d-kube-api-access-2kl7n\") pod \"payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5\" (UID: \"f6d0590b-d31b-41e8-9e9c-333030a3a96d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5"
Sep 10 04:54:53.364418 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:53.364385    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5"
Sep 10 04:54:53.376567 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:53.376537    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:53.478244 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:53.478207    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:53.549893 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:53.549859    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:53.633149 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:53.633125    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5"]
Sep 10 04:54:53.635482 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:54:53.635457    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d0590b_d31b_41e8_9e9c_333030a3a96d.slice/crio-ac331ed90d72480e6256de562be96158533ee81ecac17100b488d252572a304d WatchSource:0}: Error finding container ac331ed90d72480e6256de562be96158533ee81ecac17100b488d252572a304d: Status 404 returned error can't find the container with id ac331ed90d72480e6256de562be96158533ee81ecac17100b488d252572a304d
Sep 10 04:54:53.939140 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:53.939031    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:54.022418 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.022375    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:54.029271 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.029243    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c"]
Sep 10 04:54:54.033151 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.033129    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c"
Sep 10 04:54:54.036791 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.036769    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-13af27-dockercfg-r4zq2\""
Sep 10 04:54:54.036923 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.036772    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-13af27\""
Sep 10 04:54:54.049056 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.049029    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c"]
Sep 10 04:54:54.071570 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.071546    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck"]
Sep 10 04:54:54.074738 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.074719    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck"
Sep 10 04:54:54.098343 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.098316    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck"]
Sep 10 04:54:54.120679 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.120644    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5" event={"ID":"f6d0590b-d31b-41e8-9e9c-333030a3a96d","Type":"ContainerStarted","Data":"25a07260b665c5157b95d6490090ca397d0dcaaa6ec2cc3c98d9dce5b31c4eed"}
Sep 10 04:54:54.120838 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.120716    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5" event={"ID":"f6d0590b-d31b-41e8-9e9c-333030a3a96d","Type":"ContainerStarted","Data":"ac331ed90d72480e6256de562be96158533ee81ecac17100b488d252572a304d"}
Sep 10 04:54:54.120838 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.120747    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5"
Sep 10 04:54:54.130045 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.130020    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:54.141854 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.141810    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5" podStartSLOduration=1.141792919 podStartE2EDuration="1.141792919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:54:54.138882457 +0000 UTC m=+830.900651665" watchObservedRunningTime="2026-09-10 04:54:54.141792919 +0000 UTC m=+830.903562102"
Sep 10 04:54:54.186865 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.186840    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlw4w\" (UniqueName: \"kubernetes.io/projected/96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd-kube-api-access-qlw4w\") pod \"payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck\" (UID: \"96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck"
Sep 10 04:54:54.186984 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.186877    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88265117-d02a-4700-9d36-d9486f16067e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c\" (UID: \"88265117-d02a-4700-9d36-d9486f16067e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c"
Sep 10 04:54:54.186984 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.186948    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck\" (UID: \"96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck"
Sep 10 04:54:54.187091 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.186992    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn6hw\" (UniqueName: \"kubernetes.io/projected/88265117-d02a-4700-9d36-d9486f16067e-kube-api-access-zn6hw\") pod \"payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c\" (UID: \"88265117-d02a-4700-9d36-d9486f16067e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c"
Sep 10 04:54:54.230257 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.230232    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:54.287656 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.287632    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck\" (UID: \"96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck"
Sep 10 04:54:54.287760 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.287668    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn6hw\" (UniqueName: \"kubernetes.io/projected/88265117-d02a-4700-9d36-d9486f16067e-kube-api-access-zn6hw\") pod \"payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c\" (UID: \"88265117-d02a-4700-9d36-d9486f16067e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c"
Sep 10 04:54:54.287760 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.287704    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlw4w\" (UniqueName: \"kubernetes.io/projected/96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd-kube-api-access-qlw4w\") pod \"payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck\" (UID: \"96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck"
Sep 10 04:54:54.287760 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.287736    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88265117-d02a-4700-9d36-d9486f16067e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c\" (UID: \"88265117-d02a-4700-9d36-d9486f16067e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c"
Sep 10 04:54:54.288601 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.288579    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck\" (UID: \"96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck"
Sep 10 04:54:54.288664 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.288579    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88265117-d02a-4700-9d36-d9486f16067e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c\" (UID: \"88265117-d02a-4700-9d36-d9486f16067e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c"
Sep 10 04:54:54.295409 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.295382    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn6hw\" (UniqueName: \"kubernetes.io/projected/88265117-d02a-4700-9d36-d9486f16067e-kube-api-access-zn6hw\") pod \"payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c\" (UID: \"88265117-d02a-4700-9d36-d9486f16067e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c"
Sep 10 04:54:54.295498 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.295422    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlw4w\" (UniqueName: \"kubernetes.io/projected/96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd-kube-api-access-qlw4w\") pod \"payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck\" (UID: \"96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck"
Sep 10 04:54:54.344209 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.344184    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c"
Sep 10 04:54:54.384184 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.384150    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck"
Sep 10 04:54:54.427660 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.427617    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:54.617769 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.617746    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c"]
Sep 10 04:54:54.619680 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:54:54.619651    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88265117_d02a_4700_9d36_d9486f16067e.slice/crio-6abf6342309ed9c5ea9bfd3620aab6b314021b5cf067d18f9fe5c53e74a33444 WatchSource:0}: Error finding container 6abf6342309ed9c5ea9bfd3620aab6b314021b5cf067d18f9fe5c53e74a33444: Status 404 returned error can't find the container with id 6abf6342309ed9c5ea9bfd3620aab6b314021b5cf067d18f9fe5c53e74a33444
Sep 10 04:54:54.665432 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:54.665413    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck"]
Sep 10 04:54:54.670055 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:54:54.670030    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cbdc76_0f4f_4f5b_bf3d_6bb4650860bd.slice/crio-b5b275c5f4706a9d6eba05238c2f66fb6fc0ff3c3af2e971c5c1eeeee6494342 WatchSource:0}: Error finding container b5b275c5f4706a9d6eba05238c2f66fb6fc0ff3c3af2e971c5c1eeeee6494342: Status 404 returned error can't find the container with id b5b275c5f4706a9d6eba05238c2f66fb6fc0ff3c3af2e971c5c1eeeee6494342
Sep 10 04:54:55.127440 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:55.127408    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c" event={"ID":"88265117-d02a-4700-9d36-d9486f16067e","Type":"ContainerStarted","Data":"ddf2ebd23b7d443b4a5fd04d9950d9bbfd7c2415c9f529dd5f1218ed68ee9eac"}
Sep 10 04:54:55.127638 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:55.127460    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c" event={"ID":"88265117-d02a-4700-9d36-d9486f16067e","Type":"ContainerStarted","Data":"6abf6342309ed9c5ea9bfd3620aab6b314021b5cf067d18f9fe5c53e74a33444"}
Sep 10 04:54:55.127638 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:55.127501    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c"
Sep 10 04:54:55.129300 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:55.129265    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck" event={"ID":"96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd","Type":"ContainerStarted","Data":"d37c9574670889ddb0fca76b7c0e8162715b93633fab68c6754adc5c71419294"}
Sep 10 04:54:55.129427 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:55.129307    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck" event={"ID":"96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd","Type":"ContainerStarted","Data":"b5b275c5f4706a9d6eba05238c2f66fb6fc0ff3c3af2e971c5c1eeeee6494342"}
Sep 10 04:54:55.143315 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:55.143263    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c" podStartSLOduration=1.143246374 podStartE2EDuration="1.143246374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:54:55.142038487 +0000 UTC m=+831.903807670" watchObservedRunningTime="2026-09-10 04:54:55.143246374 +0000 UTC m=+831.905015558"
Sep 10 04:54:55.159040 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:55.158995    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck" podStartSLOduration=1.158985163 podStartE2EDuration="1.158985163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:54:55.15778448 +0000 UTC m=+831.919553664" watchObservedRunningTime="2026-09-10 04:54:55.158985163 +0000 UTC m=+831.920754344"
Sep 10 04:54:55.236153 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:55.236113    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:55.966494 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:55.966447    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:56.100184 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:56.100147    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:56.134499 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:56.134465    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck"
Sep 10 04:54:56.177024 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:56.176982    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:56.277085 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:56.276962    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:56.377292 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:56.377255    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:56.533060 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:56.532975    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:56.636947 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:56.636907    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:56.727237 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:56.727205    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656"]
Sep 10 04:54:56.729510 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:56.729415    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:56.730767 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:56.730745    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656"
Sep 10 04:54:56.733248 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:56.733228    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f3cc69\""
Sep 10 04:54:56.733356 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:56.733228    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f3cc69-dockercfg-g2snh\""
Sep 10 04:54:56.740673 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:56.740652    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656"]
Sep 10 04:54:56.761099 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:56.761075    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp"]
Sep 10 04:54:56.764164 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:56.764149    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp"
Sep 10 04:54:56.775091 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:56.775054    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp"]
Sep 10 04:54:56.805904 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:56.805831    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:56.910897 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:56.910866    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656\" (UID: \"5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656"
Sep 10 04:54:56.910897 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:56.910900    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e5151f-e68c-4367-8315-a47724341cfd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp\" (UID: \"13e5151f-e68c-4367-8315-a47724341cfd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp"
Sep 10 04:54:56.911112 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:56.911022    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpr2v\" (UniqueName: \"kubernetes.io/projected/5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6-kube-api-access-wpr2v\") pod \"payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656\" (UID: \"5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656"
Sep 10 04:54:56.911112 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:56.911076    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2ttf\" (UniqueName: \"kubernetes.io/projected/13e5151f-e68c-4367-8315-a47724341cfd-kube-api-access-n2ttf\") pod \"payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp\" (UID: \"13e5151f-e68c-4367-8315-a47724341cfd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp"
Sep 10 04:54:57.018230 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:57.012798    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpr2v\" (UniqueName: \"kubernetes.io/projected/5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6-kube-api-access-wpr2v\") pod \"payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656\" (UID: \"5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656"
Sep 10 04:54:57.018230 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:57.012943    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2ttf\" (UniqueName: \"kubernetes.io/projected/13e5151f-e68c-4367-8315-a47724341cfd-kube-api-access-n2ttf\") pod \"payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp\" (UID: \"13e5151f-e68c-4367-8315-a47724341cfd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp"
Sep 10 04:54:57.018230 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:57.013016    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656\" (UID: \"5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656"
Sep 10 04:54:57.018230 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:57.013059    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e5151f-e68c-4367-8315-a47724341cfd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp\" (UID: \"13e5151f-e68c-4367-8315-a47724341cfd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp"
Sep 10 04:54:57.018230 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:57.014729    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656\" (UID: \"5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656"
Sep 10 04:54:57.018230 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:57.014743    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e5151f-e68c-4367-8315-a47724341cfd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp\" (UID: \"13e5151f-e68c-4367-8315-a47724341cfd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp"
Sep 10 04:54:57.021305 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:57.021285    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpr2v\" (UniqueName: \"kubernetes.io/projected/5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6-kube-api-access-wpr2v\") pod \"payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656\" (UID: \"5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656"
Sep 10 04:54:57.021553 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:57.021535    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2ttf\" (UniqueName: \"kubernetes.io/projected/13e5151f-e68c-4367-8315-a47724341cfd-kube-api-access-n2ttf\") pod \"payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp\" (UID: \"13e5151f-e68c-4367-8315-a47724341cfd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp"
Sep 10 04:54:57.042467 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:57.042442    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656"
Sep 10 04:54:57.073617 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:57.073547    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp"
Sep 10 04:54:57.217545 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:57.217506    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:57.315847 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:57.315822    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656"]
Sep 10 04:54:57.317702 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:54:57.317680    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a29bfa1_f975_4d5f_be6e_6c4a3db78cf6.slice/crio-bb6adefd12fc0fabc3b77028a6a642f830bda9174809b66f0adecd895aa3cbe9 WatchSource:0}: Error finding container bb6adefd12fc0fabc3b77028a6a642f830bda9174809b66f0adecd895aa3cbe9: Status 404 returned error can't find the container with id bb6adefd12fc0fabc3b77028a6a642f830bda9174809b66f0adecd895aa3cbe9
Sep 10 04:54:57.354501 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:57.354472    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp"]
Sep 10 04:54:57.356793 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:54:57.356755    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e5151f_e68c_4367_8315_a47724341cfd.slice/crio-177b058964a0e52af1284711034a445e9c277baa417cd843ee58b985a05587d3 WatchSource:0}: Error finding container 177b058964a0e52af1284711034a445e9c277baa417cd843ee58b985a05587d3: Status 404 returned error can't find the container with id 177b058964a0e52af1284711034a445e9c277baa417cd843ee58b985a05587d3
Sep 10 04:54:57.522607 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:57.522560    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:58.148794 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:58.148756    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656" event={"ID":"5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6","Type":"ContainerStarted","Data":"a73a8877db1876ed55fb5f4c82796ebee941d1f8ea1db4aeebaa0cb1e9141556"}
Sep 10 04:54:58.149313 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:58.148825    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656" event={"ID":"5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6","Type":"ContainerStarted","Data":"bb6adefd12fc0fabc3b77028a6a642f830bda9174809b66f0adecd895aa3cbe9"}
Sep 10 04:54:58.149313 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:58.148863    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656"
Sep 10 04:54:58.150516 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:58.150490    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp" event={"ID":"13e5151f-e68c-4367-8315-a47724341cfd","Type":"ContainerStarted","Data":"d9fb0d736b6a979f8e9e9ffd748e644a36dafa2a8ffa076f94f693712d2a0f37"}
Sep 10 04:54:58.150646 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:58.150537    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp" event={"ID":"13e5151f-e68c-4367-8315-a47724341cfd","Type":"ContainerStarted","Data":"177b058964a0e52af1284711034a445e9c277baa417cd843ee58b985a05587d3"}
Sep 10 04:54:58.150646 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:58.150573    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp"
Sep 10 04:54:58.167323 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:58.167265    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656" podStartSLOduration=2.167242942 podStartE2EDuration="2.167242942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:54:58.165411703 +0000 UTC m=+834.927180886" watchObservedRunningTime="2026-09-10 04:54:58.167242942 +0000 UTC m=+834.929012125"
Sep 10 04:54:58.185002 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:58.184942    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp" podStartSLOduration=2.184924963 podStartE2EDuration="2.184924963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:54:58.184122693 +0000 UTC m=+834.945891875" watchObservedRunningTime="2026-09-10 04:54:58.184924963 +0000 UTC m=+834.946694192"
Sep 10 04:54:58.226634 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:58.226597    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:58.709405 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:58.709364    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:54:58.800893 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:54:58.800848    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:00.028863 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:00.028822    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:00.126637 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:00.126589    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:00.555658 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:00.555622    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:00.654199 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:00.654161    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:00.721468 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:00.721430    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6"]
Sep 10 04:55:00.726797 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:00.726743    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:00.727479 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:00.727454    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6"
Sep 10 04:55:00.730260 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:00.730237    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e79d9e97\""
Sep 10 04:55:00.730477 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:00.730325    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e79d9e97-dockercfg-n9pzp\""
Sep 10 04:55:00.739339 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:00.739168    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6"]
Sep 10 04:55:00.773289 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:00.773251    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf"]
Sep 10 04:55:00.776638 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:00.776613    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf"
Sep 10 04:55:00.784889 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:00.784863    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf"]
Sep 10 04:55:00.848530 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:00.848451    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q6ln\" (UniqueName: \"kubernetes.io/projected/78890332-ce3e-48b9-a54c-0ebf0a8e0b59-kube-api-access-4q6ln\") pod \"payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6\" (UID: \"78890332-ce3e-48b9-a54c-0ebf0a8e0b59\") " pod="openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6"
Sep 10 04:55:00.848669 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:00.848561    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78890332-ce3e-48b9-a54c-0ebf0a8e0b59-plugins-config-volume\") pod \"payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6\" (UID: \"78890332-ce3e-48b9-a54c-0ebf0a8e0b59\") " pod="openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6"
Sep 10 04:55:00.949412 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:00.949376    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78890332-ce3e-48b9-a54c-0ebf0a8e0b59-plugins-config-volume\") pod \"payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6\" (UID: \"78890332-ce3e-48b9-a54c-0ebf0a8e0b59\") " pod="openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6"
Sep 10 04:55:00.949560 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:00.949431    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z44r\" (UniqueName: \"kubernetes.io/projected/187c339f-5bb7-4e62-acb1-23b8f221528c-kube-api-access-5z44r\") pod \"payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf\" (UID: \"187c339f-5bb7-4e62-acb1-23b8f221528c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf"
Sep 10 04:55:00.949605 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:00.949554    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187c339f-5bb7-4e62-acb1-23b8f221528c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf\" (UID: \"187c339f-5bb7-4e62-acb1-23b8f221528c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf"
Sep 10 04:55:00.949650 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:00.949598    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q6ln\" (UniqueName: \"kubernetes.io/projected/78890332-ce3e-48b9-a54c-0ebf0a8e0b59-kube-api-access-4q6ln\") pod \"payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6\" (UID: \"78890332-ce3e-48b9-a54c-0ebf0a8e0b59\") " pod="openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6"
Sep 10 04:55:00.950283 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:00.950263    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78890332-ce3e-48b9-a54c-0ebf0a8e0b59-plugins-config-volume\") pod \"payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6\" (UID: \"78890332-ce3e-48b9-a54c-0ebf0a8e0b59\") " pod="openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6"
Sep 10 04:55:00.957683 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:00.957652    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q6ln\" (UniqueName: \"kubernetes.io/projected/78890332-ce3e-48b9-a54c-0ebf0a8e0b59-kube-api-access-4q6ln\") pod \"payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6\" (UID: \"78890332-ce3e-48b9-a54c-0ebf0a8e0b59\") " pod="openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6"
Sep 10 04:55:01.046028 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:01.045993    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6"
Sep 10 04:55:01.050143 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:01.050112    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187c339f-5bb7-4e62-acb1-23b8f221528c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf\" (UID: \"187c339f-5bb7-4e62-acb1-23b8f221528c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf"
Sep 10 04:55:01.050253 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:01.050226    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z44r\" (UniqueName: \"kubernetes.io/projected/187c339f-5bb7-4e62-acb1-23b8f221528c-kube-api-access-5z44r\") pod \"payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf\" (UID: \"187c339f-5bb7-4e62-acb1-23b8f221528c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf"
Sep 10 04:55:01.050772 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:01.050746    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187c339f-5bb7-4e62-acb1-23b8f221528c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf\" (UID: \"187c339f-5bb7-4e62-acb1-23b8f221528c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf"
Sep 10 04:55:01.058310 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:01.058277    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z44r\" (UniqueName: \"kubernetes.io/projected/187c339f-5bb7-4e62-acb1-23b8f221528c-kube-api-access-5z44r\") pod \"payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf\" (UID: \"187c339f-5bb7-4e62-acb1-23b8f221528c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf"
Sep 10 04:55:01.087927 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:01.087879    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf"
Sep 10 04:55:01.148568 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:01.146285    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:01.243991 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:01.243950    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:01.332990 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:01.332964    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6"]
Sep 10 04:55:01.334503 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:55:01.334474    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78890332_ce3e_48b9_a54c_0ebf0a8e0b59.slice/crio-6e3a318e9ba3d8aba0c1f57d26988c58bf3fd8e0dc6c257b00b8bb3e662fa997 WatchSource:0}: Error finding container 6e3a318e9ba3d8aba0c1f57d26988c58bf3fd8e0dc6c257b00b8bb3e662fa997: Status 404 returned error can't find the container with id 6e3a318e9ba3d8aba0c1f57d26988c58bf3fd8e0dc6c257b00b8bb3e662fa997
Sep 10 04:55:01.397271 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:01.397246    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf"]
Sep 10 04:55:01.399030 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:55:01.399002    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod187c339f_5bb7_4e62_acb1_23b8f221528c.slice/crio-eacccc93bf5fe405d764ba5839bac0711051b89bd872cc0673333a6facef6741 WatchSource:0}: Error finding container eacccc93bf5fe405d764ba5839bac0711051b89bd872cc0673333a6facef6741: Status 404 returned error can't find the container with id eacccc93bf5fe405d764ba5839bac0711051b89bd872cc0673333a6facef6741
Sep 10 04:55:02.173391 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:02.173354    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf" event={"ID":"187c339f-5bb7-4e62-acb1-23b8f221528c","Type":"ContainerStarted","Data":"1624a6ea077b85c72ed70a7debecb4eed26cf4c997264e86a462aa9c725b42aa"}
Sep 10 04:55:02.173879 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:02.173416    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf" event={"ID":"187c339f-5bb7-4e62-acb1-23b8f221528c","Type":"ContainerStarted","Data":"eacccc93bf5fe405d764ba5839bac0711051b89bd872cc0673333a6facef6741"}
Sep 10 04:55:02.173879 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:02.173475    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf"
Sep 10 04:55:02.175060 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:02.175033    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6" event={"ID":"78890332-ce3e-48b9-a54c-0ebf0a8e0b59","Type":"ContainerStarted","Data":"efbc58100ce7dad19d50ae7328ea272f576da67a73cb8a824a18e14af712633e"}
Sep 10 04:55:02.175203 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:02.175097    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6" event={"ID":"78890332-ce3e-48b9-a54c-0ebf0a8e0b59","Type":"ContainerStarted","Data":"6e3a318e9ba3d8aba0c1f57d26988c58bf3fd8e0dc6c257b00b8bb3e662fa997"}
Sep 10 04:55:02.175203 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:02.175120    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6"
Sep 10 04:55:02.193333 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:02.193282    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf" podStartSLOduration=2.193267209 podStartE2EDuration="2.193267209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:55:02.191243316 +0000 UTC m=+838.953012499" watchObservedRunningTime="2026-09-10 04:55:02.193267209 +0000 UTC m=+838.955036391"
Sep 10 04:55:02.210127 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:02.210058    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6" podStartSLOduration=2.210040866 podStartE2EDuration="2.210040866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:55:02.208886242 +0000 UTC m=+838.970655425" watchObservedRunningTime="2026-09-10 04:55:02.210040866 +0000 UTC m=+838.971810049"
Sep 10 04:55:02.444418 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:02.444330    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:02.546011 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:02.545974    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:02.958926 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:02.958894    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:03.060887 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:03.060855    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:03.150416 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:03.150380    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:05.131108 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:05.131059    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5"
Sep 10 04:55:06.137294 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:06.137251    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c"
Sep 10 04:55:07.143422 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:07.143392    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck"
Sep 10 04:55:08.292254 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:08.292215    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:08.457486 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:08.457442    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:08.558667 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:08.558586    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:08.657641 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:08.657601    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:09.157079 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:09.157036    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp"
Sep 10 04:55:09.157240 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:09.157219    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656"
Sep 10 04:55:11.395238 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:11.395203    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:13.034375 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:13.034338    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:13.181926 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:13.181895    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6"
Sep 10 04:55:13.182112 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:13.182002    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf"
Sep 10 04:55:13.374825 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:13.374735    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:14.933089 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:14.933032    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:15.834983 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:15.834951    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:16.197941 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:16.197905    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:16.300013 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:16.299977    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:16.735651 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:16.735617    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:16.819824 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:16.819779    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:16.948236 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:16.948199    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:17.631714 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:17.631676    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:17.898167 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:17.898054    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:17.997449 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:17.997415    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:18.131536 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:18.131493    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:19.027558 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:19.027520    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:19.341675 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:19.341587    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:19.646935 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:19.646849    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:20.160785 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:20.160752    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:20.983405 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:20.983358    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:23.209649 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:23.209603    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:23.303480 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:23.303439    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:24.394977 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:24.394940    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:24.417587 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:24.417556    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79d9e97-g8fk4"]
Sep 10 04:55:24.422521 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:24.422501    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79d9e97-g8fk4"
Sep 10 04:55:24.425741 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:24.425720    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dkf6w\""
Sep 10 04:55:24.425810 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:24.425767    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 04:55:24.434986 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:24.434964    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79d9e97-g8fk4"]
Sep 10 04:55:24.542096 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:24.542051    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae79eff4-38e5-4143-a130-f168d1635271-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e79d9e97-g8fk4\" (UID: \"ae79eff4-38e5-4143-a130-f168d1635271\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79d9e97-g8fk4"
Sep 10 04:55:24.584984 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:24.584952    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:24.642502 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:24.642471    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae79eff4-38e5-4143-a130-f168d1635271-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e79d9e97-g8fk4\" (UID: \"ae79eff4-38e5-4143-a130-f168d1635271\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79d9e97-g8fk4"
Sep 10 04:55:24.643089 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:24.643047    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae79eff4-38e5-4143-a130-f168d1635271-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e79d9e97-g8fk4\" (UID: \"ae79eff4-38e5-4143-a130-f168d1635271\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79d9e97-g8fk4"
Sep 10 04:55:24.663133 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:24.663033    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:24.731873 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:24.731842    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79d9e97-g8fk4"
Sep 10 04:55:24.766794 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:24.766760    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:24.998930 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:24.998892    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79d9e97-g8fk4"]
Sep 10 04:55:25.001023 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:55:25.000987    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae79eff4_38e5_4143_a130_f168d1635271.slice/crio-8c59ef7c9a7ca9fb201a7cadb4a9028046337a7b73df64182ee291bd7b3c69ef WatchSource:0}: Error finding container 8c59ef7c9a7ca9fb201a7cadb4a9028046337a7b73df64182ee291bd7b3c69ef: Status 404 returned error can't find the container with id 8c59ef7c9a7ca9fb201a7cadb4a9028046337a7b73df64182ee291bd7b3c69ef
Sep 10 04:55:25.160967 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.160926    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:25.262350 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.262265    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:25.296159 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.296123    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79d9e97-g8fk4" event={"ID":"ae79eff4-38e5-4143-a130-f168d1635271","Type":"ContainerStarted","Data":"8c59ef7c9a7ca9fb201a7cadb4a9028046337a7b73df64182ee291bd7b3c69ef"}
Sep 10 04:55:25.362974 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.362941    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:25.647362 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.647277    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:25.730273 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.730238    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:25.754012 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.753981    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g"]
Sep 10 04:55:25.758585 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.758566    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g"
Sep 10 04:55:25.760977 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.760956    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c9a840\""
Sep 10 04:55:25.768279 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.768257    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g"]
Sep 10 04:55:25.785671 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.785650    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7"]
Sep 10 04:55:25.789105 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.789084    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7"
Sep 10 04:55:25.791690 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.791674    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c9a840-dockercfg-cj2fd\""
Sep 10 04:55:25.801049 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.801021    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7"]
Sep 10 04:55:25.853171 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.853142    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwf8s\" (UniqueName: \"kubernetes.io/projected/024dc2ea-998d-459d-bd00-2f2a054ae43e-kube-api-access-pwf8s\") pod \"payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7\" (UID: \"024dc2ea-998d-459d-bd00-2f2a054ae43e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7"
Sep 10 04:55:25.853313 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.853184    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024dc2ea-998d-459d-bd00-2f2a054ae43e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7\" (UID: \"024dc2ea-998d-459d-bd00-2f2a054ae43e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7"
Sep 10 04:55:25.853313 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.853205    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g\" (UID: \"b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g"
Sep 10 04:55:25.853423 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.853320    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcgks\" (UniqueName: \"kubernetes.io/projected/b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b-kube-api-access-kcgks\") pod \"payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g\" (UID: \"b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g"
Sep 10 04:55:25.954281 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.954242    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcgks\" (UniqueName: \"kubernetes.io/projected/b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b-kube-api-access-kcgks\") pod \"payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g\" (UID: \"b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g"
Sep 10 04:55:25.954449 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.954363    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwf8s\" (UniqueName: \"kubernetes.io/projected/024dc2ea-998d-459d-bd00-2f2a054ae43e-kube-api-access-pwf8s\") pod \"payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7\" (UID: \"024dc2ea-998d-459d-bd00-2f2a054ae43e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7"
Sep 10 04:55:25.961325 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.954701    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024dc2ea-998d-459d-bd00-2f2a054ae43e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7\" (UID: \"024dc2ea-998d-459d-bd00-2f2a054ae43e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7"
Sep 10 04:55:25.961325 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.955031    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g\" (UID: \"b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g"
Sep 10 04:55:25.961325 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.955868    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024dc2ea-998d-459d-bd00-2f2a054ae43e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7\" (UID: \"024dc2ea-998d-459d-bd00-2f2a054ae43e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7"
Sep 10 04:55:25.961325 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.955970    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g\" (UID: \"b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g"
Sep 10 04:55:25.963961 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.963936    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwf8s\" (UniqueName: \"kubernetes.io/projected/024dc2ea-998d-459d-bd00-2f2a054ae43e-kube-api-access-pwf8s\") pod \"payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7\" (UID: \"024dc2ea-998d-459d-bd00-2f2a054ae43e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7"
Sep 10 04:55:25.964090 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:25.963959    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcgks\" (UniqueName: \"kubernetes.io/projected/b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b-kube-api-access-kcgks\") pod \"payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g\" (UID: \"b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g"
Sep 10 04:55:26.049060 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:26.049015    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:26.070847 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:26.070818    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g"
Sep 10 04:55:26.098212 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:26.098184    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7"
Sep 10 04:55:26.354226 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:26.354140    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g"]
Sep 10 04:55:26.358058 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:55:26.358022    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34a5f6d_c5bf_4859_be36_c77d5d7f6f0b.slice/crio-f8d1fd410cc9c69632a64c5bf10c79948a7e916a1db74948032a265cd054738f WatchSource:0}: Error finding container f8d1fd410cc9c69632a64c5bf10c79948a7e916a1db74948032a265cd054738f: Status 404 returned error can't find the container with id f8d1fd410cc9c69632a64c5bf10c79948a7e916a1db74948032a265cd054738f
Sep 10 04:55:26.393050 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:26.393016    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7"]
Sep 10 04:55:26.395637 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:55:26.395610    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024dc2ea_998d_459d_bd00_2f2a054ae43e.slice/crio-8b0cee62cbfbf6e2260a66e779fd589fbd4025626692c009505ca40349579122 WatchSource:0}: Error finding container 8b0cee62cbfbf6e2260a66e779fd589fbd4025626692c009505ca40349579122: Status 404 returned error can't find the container with id 8b0cee62cbfbf6e2260a66e779fd589fbd4025626692c009505ca40349579122
Sep 10 04:55:26.957116 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:26.957032    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:27.309486 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:27.309397    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7" event={"ID":"024dc2ea-998d-459d-bd00-2f2a054ae43e","Type":"ContainerStarted","Data":"4237ceb6dcd8a5393248c5cc04d7a7c7dad298e4c28408d02f3c3361ca9bc687"}
Sep 10 04:55:27.309486 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:27.309463    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7"
Sep 10 04:55:27.309486 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:27.309487    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7" event={"ID":"024dc2ea-998d-459d-bd00-2f2a054ae43e","Type":"ContainerStarted","Data":"8b0cee62cbfbf6e2260a66e779fd589fbd4025626692c009505ca40349579122"}
Sep 10 04:55:27.311027 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:27.310993    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g" event={"ID":"b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b","Type":"ContainerStarted","Data":"906085c0a1b45309b424907628234f8ec8182229cc3dcf5d6ff5903a04927965"}
Sep 10 04:55:27.311146 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:27.311037    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g" event={"ID":"b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b","Type":"ContainerStarted","Data":"f8d1fd410cc9c69632a64c5bf10c79948a7e916a1db74948032a265cd054738f"}
Sep 10 04:55:27.311146 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:27.311059    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g"
Sep 10 04:55:27.325663 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:27.325622    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7" podStartSLOduration=2.3256068389999998 podStartE2EDuration="2.325606839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:55:27.324196475 +0000 UTC m=+864.085965658" watchObservedRunningTime="2026-09-10 04:55:27.325606839 +0000 UTC m=+864.087376073"
Sep 10 04:55:27.338910 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:27.338851    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g" podStartSLOduration=2.3388333230000002 podStartE2EDuration="2.338833323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:55:27.338561789 +0000 UTC m=+864.100330972" watchObservedRunningTime="2026-09-10 04:55:27.338833323 +0000 UTC m=+864.100602505"
Sep 10 04:55:27.644130 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:27.644099    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:28.318990 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:28.318891    2575 generic.go:356] "Generic (PLEG): container finished" podID="ae79eff4-38e5-4143-a130-f168d1635271" containerID="7ceefdf542739b0f748d40cfb38f8795944a03eee6ef521384ade78b1a82441b" exitCode=0
Sep 10 04:55:28.319450 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:28.318983    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79d9e97-g8fk4" event={"ID":"ae79eff4-38e5-4143-a130-f168d1635271","Type":"ContainerDied","Data":"7ceefdf542739b0f748d40cfb38f8795944a03eee6ef521384ade78b1a82441b"}
Sep 10 04:55:28.561477 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:28.561439    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:29.461725 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:29.461703    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79d9e97-g8fk4"
Sep 10 04:55:29.588711 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:29.588614    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae79eff4-38e5-4143-a130-f168d1635271-maas-api-service-ca\") pod \"ae79eff4-38e5-4143-a130-f168d1635271\" (UID: \"ae79eff4-38e5-4143-a130-f168d1635271\") "
Sep 10 04:55:29.588998 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:29.588972    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae79eff4-38e5-4143-a130-f168d1635271-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ae79eff4-38e5-4143-a130-f168d1635271" (UID: "ae79eff4-38e5-4143-a130-f168d1635271"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 04:55:29.689839 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:29.689799    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae79eff4-38e5-4143-a130-f168d1635271-maas-api-service-ca\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:55:30.333333 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:30.333301    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79d9e97-g8fk4"
Sep 10 04:55:30.333514 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:30.333330    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79d9e97-g8fk4" event={"ID":"ae79eff4-38e5-4143-a130-f168d1635271","Type":"ContainerDied","Data":"8c59ef7c9a7ca9fb201a7cadb4a9028046337a7b73df64182ee291bd7b3c69ef"}
Sep 10 04:55:30.333514 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:30.333375    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c59ef7c9a7ca9fb201a7cadb4a9028046337a7b73df64182ee291bd7b3c69ef"
Sep 10 04:55:30.362624 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:30.362596    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:30.465531 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:30.465488    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:31.174109 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:31.174044    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:38.320911 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:38.320849    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7"
Sep 10 04:55:38.321397 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:38.320946    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g"
Sep 10 04:55:40.103491 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:40.103454    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:40.669522 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:40.669488    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:40.697229 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:40.697197    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6"]
Sep 10 04:55:40.697438 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:40.697412    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6" podUID="78890332-ce3e-48b9-a54c-0ebf0a8e0b59" containerName="payload-processing" containerID="cri-o://efbc58100ce7dad19d50ae7328ea272f576da67a73cb8a824a18e14af712633e" gracePeriod=30
Sep 10 04:55:40.705058 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:40.705032    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf"]
Sep 10 04:55:40.705301 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:40.705275    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf" podUID="187c339f-5bb7-4e62-acb1-23b8f221528c" containerName="payload-pre-processing" containerID="cri-o://1624a6ea077b85c72ed70a7debecb4eed26cf4c997264e86a462aa9c725b42aa" gracePeriod=30
Sep 10 04:55:40.737468 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:40.737419    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:40.798870 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:40.798835    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:40.854322 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:40.854284    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79d9e97-g8fk4"]
Sep 10 04:55:40.858571 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:40.858541    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79d9e97-g8fk4"]
Sep 10 04:55:40.860578 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:40.860549    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:40.927519 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:40.927415    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:40.983309 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:40.983280    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6"
Sep 10 04:55:40.986764 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:40.986743    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf"
Sep 10 04:55:41.007756 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.007712    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:41.099975 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.099925    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187c339f-5bb7-4e62-acb1-23b8f221528c-plugins-config-volume\") pod \"187c339f-5bb7-4e62-acb1-23b8f221528c\" (UID: \"187c339f-5bb7-4e62-acb1-23b8f221528c\") "
Sep 10 04:55:41.100200 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.100039    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78890332-ce3e-48b9-a54c-0ebf0a8e0b59-plugins-config-volume\") pod \"78890332-ce3e-48b9-a54c-0ebf0a8e0b59\" (UID: \"78890332-ce3e-48b9-a54c-0ebf0a8e0b59\") "
Sep 10 04:55:41.100200 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.100124    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5z44r\" (UniqueName: \"kubernetes.io/projected/187c339f-5bb7-4e62-acb1-23b8f221528c-kube-api-access-5z44r\") pod \"187c339f-5bb7-4e62-acb1-23b8f221528c\" (UID: \"187c339f-5bb7-4e62-acb1-23b8f221528c\") "
Sep 10 04:55:41.100200 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.100164    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4q6ln\" (UniqueName: \"kubernetes.io/projected/78890332-ce3e-48b9-a54c-0ebf0a8e0b59-kube-api-access-4q6ln\") pod \"78890332-ce3e-48b9-a54c-0ebf0a8e0b59\" (UID: \"78890332-ce3e-48b9-a54c-0ebf0a8e0b59\") "
Sep 10 04:55:41.100364 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.100298    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/187c339f-5bb7-4e62-acb1-23b8f221528c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "187c339f-5bb7-4e62-acb1-23b8f221528c" (UID: "187c339f-5bb7-4e62-acb1-23b8f221528c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 04:55:41.100449 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.100434    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187c339f-5bb7-4e62-acb1-23b8f221528c-plugins-config-volume\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:55:41.100486 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.100427    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78890332-ce3e-48b9-a54c-0ebf0a8e0b59-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "78890332-ce3e-48b9-a54c-0ebf0a8e0b59" (UID: "78890332-ce3e-48b9-a54c-0ebf0a8e0b59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 04:55:41.102433 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.102409    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/187c339f-5bb7-4e62-acb1-23b8f221528c-kube-api-access-5z44r" (OuterVolumeSpecName: "kube-api-access-5z44r") pod "187c339f-5bb7-4e62-acb1-23b8f221528c" (UID: "187c339f-5bb7-4e62-acb1-23b8f221528c"). InnerVolumeSpecName "kube-api-access-5z44r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 04:55:41.102433 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.102422    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78890332-ce3e-48b9-a54c-0ebf0a8e0b59-kube-api-access-4q6ln" (OuterVolumeSpecName: "kube-api-access-4q6ln") pod "78890332-ce3e-48b9-a54c-0ebf0a8e0b59" (UID: "78890332-ce3e-48b9-a54c-0ebf0a8e0b59"). InnerVolumeSpecName "kube-api-access-4q6ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 04:55:41.201756 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.201713    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z44r\" (UniqueName: \"kubernetes.io/projected/187c339f-5bb7-4e62-acb1-23b8f221528c-kube-api-access-5z44r\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:55:41.201756 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.201757    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q6ln\" (UniqueName: \"kubernetes.io/projected/78890332-ce3e-48b9-a54c-0ebf0a8e0b59-kube-api-access-4q6ln\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:55:41.201756 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.201767    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78890332-ce3e-48b9-a54c-0ebf0a8e0b59-plugins-config-volume\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:55:41.388344 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.388309    2575 generic.go:356] "Generic (PLEG): container finished" podID="187c339f-5bb7-4e62-acb1-23b8f221528c" containerID="1624a6ea077b85c72ed70a7debecb4eed26cf4c997264e86a462aa9c725b42aa" exitCode=0
Sep 10 04:55:41.388532 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.388356    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf"
Sep 10 04:55:41.388532 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.388394    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf" event={"ID":"187c339f-5bb7-4e62-acb1-23b8f221528c","Type":"ContainerDied","Data":"1624a6ea077b85c72ed70a7debecb4eed26cf4c997264e86a462aa9c725b42aa"}
Sep 10 04:55:41.388532 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.388469    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf" event={"ID":"187c339f-5bb7-4e62-acb1-23b8f221528c","Type":"ContainerDied","Data":"eacccc93bf5fe405d764ba5839bac0711051b89bd872cc0673333a6facef6741"}
Sep 10 04:55:41.388532 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.388505    2575 scope.go:117] "RemoveContainer" containerID="1624a6ea077b85c72ed70a7debecb4eed26cf4c997264e86a462aa9c725b42aa"
Sep 10 04:55:41.389831 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.389803    2575 generic.go:356] "Generic (PLEG): container finished" podID="78890332-ce3e-48b9-a54c-0ebf0a8e0b59" containerID="efbc58100ce7dad19d50ae7328ea272f576da67a73cb8a824a18e14af712633e" exitCode=0
Sep 10 04:55:41.389973 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.389843    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6"
Sep 10 04:55:41.389973 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.389883    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6" event={"ID":"78890332-ce3e-48b9-a54c-0ebf0a8e0b59","Type":"ContainerDied","Data":"efbc58100ce7dad19d50ae7328ea272f576da67a73cb8a824a18e14af712633e"}
Sep 10 04:55:41.389973 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.389926    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6" event={"ID":"78890332-ce3e-48b9-a54c-0ebf0a8e0b59","Type":"ContainerDied","Data":"6e3a318e9ba3d8aba0c1f57d26988c58bf3fd8e0dc6c257b00b8bb3e662fa997"}
Sep 10 04:55:41.403700 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.403675    2575 scope.go:117] "RemoveContainer" containerID="1624a6ea077b85c72ed70a7debecb4eed26cf4c997264e86a462aa9c725b42aa"
Sep 10 04:55:41.404044 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:55:41.404008    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1624a6ea077b85c72ed70a7debecb4eed26cf4c997264e86a462aa9c725b42aa\": container with ID starting with 1624a6ea077b85c72ed70a7debecb4eed26cf4c997264e86a462aa9c725b42aa not found: ID does not exist" containerID="1624a6ea077b85c72ed70a7debecb4eed26cf4c997264e86a462aa9c725b42aa"
Sep 10 04:55:41.404144 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.404042    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1624a6ea077b85c72ed70a7debecb4eed26cf4c997264e86a462aa9c725b42aa"} err="failed to get container status \"1624a6ea077b85c72ed70a7debecb4eed26cf4c997264e86a462aa9c725b42aa\": rpc error: code = NotFound desc = could not find container \"1624a6ea077b85c72ed70a7debecb4eed26cf4c997264e86a462aa9c725b42aa\": container with ID starting with 1624a6ea077b85c72ed70a7debecb4eed26cf4c997264e86a462aa9c725b42aa not found: ID does not exist"
Sep 10 04:55:41.404144 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.404060    2575 scope.go:117] "RemoveContainer" containerID="efbc58100ce7dad19d50ae7328ea272f576da67a73cb8a824a18e14af712633e"
Sep 10 04:55:41.413651 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.413631    2575 scope.go:117] "RemoveContainer" containerID="efbc58100ce7dad19d50ae7328ea272f576da67a73cb8a824a18e14af712633e"
Sep 10 04:55:41.414003 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:55:41.413985    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efbc58100ce7dad19d50ae7328ea272f576da67a73cb8a824a18e14af712633e\": container with ID starting with efbc58100ce7dad19d50ae7328ea272f576da67a73cb8a824a18e14af712633e not found: ID does not exist" containerID="efbc58100ce7dad19d50ae7328ea272f576da67a73cb8a824a18e14af712633e"
Sep 10 04:55:41.414083 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.414014    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efbc58100ce7dad19d50ae7328ea272f576da67a73cb8a824a18e14af712633e"} err="failed to get container status \"efbc58100ce7dad19d50ae7328ea272f576da67a73cb8a824a18e14af712633e\": rpc error: code = NotFound desc = could not find container \"efbc58100ce7dad19d50ae7328ea272f576da67a73cb8a824a18e14af712633e\": container with ID starting with efbc58100ce7dad19d50ae7328ea272f576da67a73cb8a824a18e14af712633e not found: ID does not exist"
Sep 10 04:55:41.425742 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.425715    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf"]
Sep 10 04:55:41.428896 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.428875    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e79d9e97-55f9bf98c8-lmwkf"]
Sep 10 04:55:41.450311 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.450272    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6"]
Sep 10 04:55:41.462850 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.462785    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e79d9e97-7948566c85-fflb6"]
Sep 10 04:55:41.537675 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.537643    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:41.663969 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.663937    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:41.740384 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.740301    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:41.864816 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.864780    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="187c339f-5bb7-4e62-acb1-23b8f221528c" path="/var/lib/kubelet/pods/187c339f-5bb7-4e62-acb1-23b8f221528c/volumes"
Sep 10 04:55:41.865209 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.865193    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78890332-ce3e-48b9-a54c-0ebf0a8e0b59" path="/var/lib/kubelet/pods/78890332-ce3e-48b9-a54c-0ebf0a8e0b59/volumes"
Sep 10 04:55:41.865491 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:41.865479    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae79eff4-38e5-4143-a130-f168d1635271" path="/var/lib/kubelet/pods/ae79eff4-38e5-4143-a130-f168d1635271/volumes"
Sep 10 04:55:42.018679 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:42.018598    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:42.581199 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:42.581161    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:44.888902 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:44.888860    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:44.990162 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:44.990126    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:45.207135 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:45.207101    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:45.312776 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:45.312742    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:45.420232 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:45.420200    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:45.786157 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:45.786115    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:45.884126 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:45.884088    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:45.924296 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:45.924264    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d"]
Sep 10 04:55:45.924648 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:45.924635    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae79eff4-38e5-4143-a130-f168d1635271" containerName="revoke-keys"
Sep 10 04:55:45.924686 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:45.924650    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae79eff4-38e5-4143-a130-f168d1635271" containerName="revoke-keys"
Sep 10 04:55:45.924686 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:45.924661    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78890332-ce3e-48b9-a54c-0ebf0a8e0b59" containerName="payload-processing"
Sep 10 04:55:45.924686 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:45.924667    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="78890332-ce3e-48b9-a54c-0ebf0a8e0b59" containerName="payload-processing"
Sep 10 04:55:45.924686 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:45.924674    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="187c339f-5bb7-4e62-acb1-23b8f221528c" containerName="payload-pre-processing"
Sep 10 04:55:45.924686 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:45.924680    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="187c339f-5bb7-4e62-acb1-23b8f221528c" containerName="payload-pre-processing"
Sep 10 04:55:45.924848 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:45.924742    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="187c339f-5bb7-4e62-acb1-23b8f221528c" containerName="payload-pre-processing"
Sep 10 04:55:45.924848 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:45.924752    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae79eff4-38e5-4143-a130-f168d1635271" containerName="revoke-keys"
Sep 10 04:55:45.924848 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:45.924760    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="78890332-ce3e-48b9-a54c-0ebf0a8e0b59" containerName="payload-processing"
Sep 10 04:55:45.930447 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:45.930427    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d"
Sep 10 04:55:45.933172 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:45.933147    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d778192d\""
Sep 10 04:55:45.933298 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:45.933149    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d778192d-dockercfg-xw46s\""
Sep 10 04:55:45.938385 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:45.938366    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d"]
Sep 10 04:55:46.042622 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:46.042539    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwfqj\" (UniqueName: \"kubernetes.io/projected/125e80df-4f04-4843-8003-436343b5b624-kube-api-access-fwfqj\") pod \"payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d\" (UID: \"125e80df-4f04-4843-8003-436343b5b624\") " pod="openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d"
Sep 10 04:55:46.042622 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:46.042593    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125e80df-4f04-4843-8003-436343b5b624-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d\" (UID: \"125e80df-4f04-4843-8003-436343b5b624\") " pod="openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d"
Sep 10 04:55:46.143607 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:46.143569    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125e80df-4f04-4843-8003-436343b5b624-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d\" (UID: \"125e80df-4f04-4843-8003-436343b5b624\") " pod="openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d"
Sep 10 04:55:46.143789 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:46.143678    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwfqj\" (UniqueName: \"kubernetes.io/projected/125e80df-4f04-4843-8003-436343b5b624-kube-api-access-fwfqj\") pod \"payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d\" (UID: \"125e80df-4f04-4843-8003-436343b5b624\") " pod="openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d"
Sep 10 04:55:46.144323 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:46.144294    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125e80df-4f04-4843-8003-436343b5b624-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d\" (UID: \"125e80df-4f04-4843-8003-436343b5b624\") " pod="openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d"
Sep 10 04:55:46.151457 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:46.151437    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwfqj\" (UniqueName: \"kubernetes.io/projected/125e80df-4f04-4843-8003-436343b5b624-kube-api-access-fwfqj\") pod \"payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d\" (UID: \"125e80df-4f04-4843-8003-436343b5b624\") " pod="openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d"
Sep 10 04:55:46.241571 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:46.241534    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d"
Sep 10 04:55:46.242935 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:46.242899    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:46.514422 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:46.514396    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d"]
Sep 10 04:55:46.516060 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:55:46.516018    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125e80df_4f04_4843_8003_436343b5b624.slice/crio-2ea98c85e677a043023ab368526acc8d5d68dbd1dc13f20d3d23363f06dee988 WatchSource:0}: Error finding container 2ea98c85e677a043023ab368526acc8d5d68dbd1dc13f20d3d23363f06dee988: Status 404 returned error can't find the container with id 2ea98c85e677a043023ab368526acc8d5d68dbd1dc13f20d3d23363f06dee988
Sep 10 04:55:47.153338 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:47.153295    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:47.423445 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:47.423354    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d" event={"ID":"125e80df-4f04-4843-8003-436343b5b624","Type":"ContainerStarted","Data":"21a5b15c2c288caf611aa30344e7b02f0aafc011bc49fe24cb704b2d3090cc1c"}
Sep 10 04:55:47.423445 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:47.423414    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d"
Sep 10 04:55:47.423445 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:47.423436    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d" event={"ID":"125e80df-4f04-4843-8003-436343b5b624","Type":"ContainerStarted","Data":"2ea98c85e677a043023ab368526acc8d5d68dbd1dc13f20d3d23363f06dee988"}
Sep 10 04:55:47.440501 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:47.440448    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d" podStartSLOduration=2.440432291 podStartE2EDuration="2.440432291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:55:47.439449867 +0000 UTC m=+884.201219049" watchObservedRunningTime="2026-09-10 04:55:47.440432291 +0000 UTC m=+884.202201470"
Sep 10 04:55:47.946879 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:47.946844    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:48.120397 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:48.120366    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:58.429863 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.429783    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d"
Sep 10 04:55:58.775963 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.775926    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"]
Sep 10 04:55:58.781825 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.781800    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:55:58.784484 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.784454    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7f2cd3\"/\"kube-root-ca.crt\""
Sep 10 04:55:58.784629 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.784454    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7f2cd3\"/\"openshift-service-ca.crt\""
Sep 10 04:55:58.785449 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.785428    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7f2cd3\"/\"default-dockercfg-q4jp4\""
Sep 10 04:55:58.785503 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.785432    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7f2cd3\"/\"simulator-w0-7f2cd3-kserve-self-signed-certs\""
Sep 10 04:55:58.789352 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.789318    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"]
Sep 10 04:55:58.860608 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.860565    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-model-cache\") pod \"simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") " pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:55:58.860608 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.860618    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxtbz\" (UniqueName: \"kubernetes.io/projected/69190cf0-aa53-48ba-81e3-f8bead942f15-kube-api-access-mxtbz\") pod \"simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") " pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:55:58.860917 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.860685    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-home\") pod \"simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") " pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:55:58.860917 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.860731    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-tmp-dir\") pod \"simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") " pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:55:58.860917 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.860748    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69190cf0-aa53-48ba-81e3-f8bead942f15-tls-certs\") pod \"simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") " pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:55:58.860917 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.860779    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-dshm\") pod \"simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") " pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:55:58.896793 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.896754    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:55:58.961215 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.961178    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-tmp-dir\") pod \"simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") " pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:55:58.961391 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.961220    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69190cf0-aa53-48ba-81e3-f8bead942f15-tls-certs\") pod \"simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") " pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:55:58.961391 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.961347    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-dshm\") pod \"simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") " pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:55:58.961510 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.961417    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-model-cache\") pod \"simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") " pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:55:58.961510 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.961437    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxtbz\" (UniqueName: \"kubernetes.io/projected/69190cf0-aa53-48ba-81e3-f8bead942f15-kube-api-access-mxtbz\") pod \"simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") " pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:55:58.961510 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.961487    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-home\") pod \"simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") " pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:55:58.961676 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.961632    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-tmp-dir\") pod \"simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") " pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:55:58.961776 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.961752    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-model-cache\") pod \"simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") " pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:55:58.961839 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.961819    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-home\") pod \"simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") " pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:55:58.963565 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.963545    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-dshm\") pod \"simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") " pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:55:58.963938 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.963918    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69190cf0-aa53-48ba-81e3-f8bead942f15-tls-certs\") pod \"simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") " pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:55:58.970106 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:58.970086    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxtbz\" (UniqueName: \"kubernetes.io/projected/69190cf0-aa53-48ba-81e3-f8bead942f15-kube-api-access-mxtbz\") pod \"simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") " pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:55:59.094773 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:59.094683    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:55:59.386912 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:59.386883    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"]
Sep 10 04:55:59.388828 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:55:59.388791    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69190cf0_aa53_48ba_81e3_f8bead942f15.slice/crio-58373ac40ded89547b889441ae32966ed47d69d9a5b8549b6c34bd21899a641a WatchSource:0}: Error finding container 58373ac40ded89547b889441ae32966ed47d69d9a5b8549b6c34bd21899a641a: Status 404 returned error can't find the container with id 58373ac40ded89547b889441ae32966ed47d69d9a5b8549b6c34bd21899a641a
Sep 10 04:55:59.484105 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:59.484042    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6" event={"ID":"69190cf0-aa53-48ba-81e3-f8bead942f15","Type":"ContainerStarted","Data":"079d38dcfa03d78921c8ec983d00ebd35f7612b3932af6f5e5de9715b729360f"}
Sep 10 04:55:59.484467 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:59.484118    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6" event={"ID":"69190cf0-aa53-48ba-81e3-f8bead942f15","Type":"ContainerStarted","Data":"58373ac40ded89547b889441ae32966ed47d69d9a5b8549b6c34bd21899a641a"}
Sep 10 04:55:59.501494 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:55:59.501450    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6" podStartSLOduration=1.5014335779999999 podStartE2EDuration="1.501433578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:55:59.50066888 +0000 UTC m=+896.262438115" watchObservedRunningTime="2026-09-10 04:55:59.501433578 +0000 UTC m=+896.263202760"
Sep 10 04:56:01.343161 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:01.343123    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:03.712416 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:03.712386    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2hd2_9d909143-4c70-4fe1-900a-3cd520db32e5/ovn-acl-logging/0.log"
Sep 10 04:56:03.825472 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:03.825445    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2hd2_9d909143-4c70-4fe1-900a-3cd520db32e5/ovn-acl-logging/0.log"
Sep 10 04:56:06.216239 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:06.216203    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:09.095770 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:09.095729    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:56:09.096197 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:09.095784    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:56:09.108409 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:09.108384    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:56:09.543857 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:09.543828    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 04:56:09.893247 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:09.893165    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d778192d-pm2v8"]
Sep 10 04:56:09.897896 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:09.897878    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d778192d-pm2v8"
Sep 10 04:56:09.900560 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:09.900535    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 04:56:09.900970 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:09.900946    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dkf6w\""
Sep 10 04:56:09.903178 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:09.903154    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d778192d-pm2v8"]
Sep 10 04:56:09.960956 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:09.960923    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7e02f4-7f48-4b86-b7f2-398de2ab6a56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d778192d-pm2v8\" (UID: \"ed7e02f4-7f48-4b86-b7f2-398de2ab6a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d778192d-pm2v8"
Sep 10 04:56:10.062376 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:10.062334    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7e02f4-7f48-4b86-b7f2-398de2ab6a56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d778192d-pm2v8\" (UID: \"ed7e02f4-7f48-4b86-b7f2-398de2ab6a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d778192d-pm2v8"
Sep 10 04:56:10.062976 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:10.062956    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7e02f4-7f48-4b86-b7f2-398de2ab6a56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d778192d-pm2v8\" (UID: \"ed7e02f4-7f48-4b86-b7f2-398de2ab6a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d778192d-pm2v8"
Sep 10 04:56:10.209544 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:10.209519    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d778192d-pm2v8"
Sep 10 04:56:10.476802 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:10.476775    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d778192d-pm2v8"]
Sep 10 04:56:10.479038 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:56:10.479008    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7e02f4_7f48_4b86_b7f2_398de2ab6a56.slice/crio-011f06cba23178044187d371355141c7d543b7104bda8e44ba5406e5e25b6345 WatchSource:0}: Error finding container 011f06cba23178044187d371355141c7d543b7104bda8e44ba5406e5e25b6345: Status 404 returned error can't find the container with id 011f06cba23178044187d371355141c7d543b7104bda8e44ba5406e5e25b6345
Sep 10 04:56:10.538466 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:10.538436    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d778192d-pm2v8" event={"ID":"ed7e02f4-7f48-4b86-b7f2-398de2ab6a56","Type":"ContainerStarted","Data":"011f06cba23178044187d371355141c7d543b7104bda8e44ba5406e5e25b6345"}
Sep 10 04:56:11.544733 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:11.544687    2575 generic.go:356] "Generic (PLEG): container finished" podID="ed7e02f4-7f48-4b86-b7f2-398de2ab6a56" containerID="ea5d794542377add928bd12429bdccc7cb3354ba556333c10e24d446082753d6" exitCode=0
Sep 10 04:56:11.545121 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:11.544768    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d778192d-pm2v8" event={"ID":"ed7e02f4-7f48-4b86-b7f2-398de2ab6a56","Type":"ContainerDied","Data":"ea5d794542377add928bd12429bdccc7cb3354ba556333c10e24d446082753d6"}
Sep 10 04:56:12.681358 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:12.681336    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d778192d-pm2v8"
Sep 10 04:56:12.786666 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:12.786632    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7e02f4-7f48-4b86-b7f2-398de2ab6a56-maas-api-service-ca\") pod \"ed7e02f4-7f48-4b86-b7f2-398de2ab6a56\" (UID: \"ed7e02f4-7f48-4b86-b7f2-398de2ab6a56\") "
Sep 10 04:56:12.787012 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:12.786991    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7e02f4-7f48-4b86-b7f2-398de2ab6a56-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ed7e02f4-7f48-4b86-b7f2-398de2ab6a56" (UID: "ed7e02f4-7f48-4b86-b7f2-398de2ab6a56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 04:56:12.887761 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:12.887680    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7e02f4-7f48-4b86-b7f2-398de2ab6a56-maas-api-service-ca\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:56:13.555026 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:13.554990    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d778192d-pm2v8"
Sep 10 04:56:13.555026 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:13.554998    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d778192d-pm2v8" event={"ID":"ed7e02f4-7f48-4b86-b7f2-398de2ab6a56","Type":"ContainerDied","Data":"011f06cba23178044187d371355141c7d543b7104bda8e44ba5406e5e25b6345"}
Sep 10 04:56:13.555257 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:13.555061    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="011f06cba23178044187d371355141c7d543b7104bda8e44ba5406e5e25b6345"
Sep 10 04:56:14.895911 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:14.895879    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:14.996611 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:14.996571    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:16.346574 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:16.346539    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:16.388485 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:16.388447    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d"]
Sep 10 04:56:16.388741 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:16.388711    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d" podUID="125e80df-4f04-4843-8003-436343b5b624" containerName="payload-processing" containerID="cri-o://21a5b15c2c288caf611aa30344e7b02f0aafc011bc49fe24cb704b2d3090cc1c" gracePeriod=30
Sep 10 04:56:16.459300 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:16.459267    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:16.556603 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:16.556560    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:16.570925 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:16.570892    2575 generic.go:356] "Generic (PLEG): container finished" podID="125e80df-4f04-4843-8003-436343b5b624" containerID="21a5b15c2c288caf611aa30344e7b02f0aafc011bc49fe24cb704b2d3090cc1c" exitCode=0
Sep 10 04:56:16.571052 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:16.570929    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d" event={"ID":"125e80df-4f04-4843-8003-436343b5b624","Type":"ContainerDied","Data":"21a5b15c2c288caf611aa30344e7b02f0aafc011bc49fe24cb704b2d3090cc1c"}
Sep 10 04:56:16.607735 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:16.607663    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d778192d-pm2v8"]
Sep 10 04:56:16.611938 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:16.611911    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d778192d-pm2v8"]
Sep 10 04:56:16.618119 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:16.618089    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:16.639235 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:16.639215    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d"
Sep 10 04:56:16.732891 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:16.732853    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwfqj\" (UniqueName: \"kubernetes.io/projected/125e80df-4f04-4843-8003-436343b5b624-kube-api-access-fwfqj\") pod \"125e80df-4f04-4843-8003-436343b5b624\" (UID: \"125e80df-4f04-4843-8003-436343b5b624\") "
Sep 10 04:56:16.733095 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:16.732918    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125e80df-4f04-4843-8003-436343b5b624-plugins-config-volume\") pod \"125e80df-4f04-4843-8003-436343b5b624\" (UID: \"125e80df-4f04-4843-8003-436343b5b624\") "
Sep 10 04:56:16.733752 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:16.733717    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125e80df-4f04-4843-8003-436343b5b624-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "125e80df-4f04-4843-8003-436343b5b624" (UID: "125e80df-4f04-4843-8003-436343b5b624"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 04:56:16.737316 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:16.737289    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125e80df-4f04-4843-8003-436343b5b624-kube-api-access-fwfqj" (OuterVolumeSpecName: "kube-api-access-fwfqj") pod "125e80df-4f04-4843-8003-436343b5b624" (UID: "125e80df-4f04-4843-8003-436343b5b624"). InnerVolumeSpecName "kube-api-access-fwfqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 04:56:16.767167 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:16.767129    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:16.834459 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:16.834424    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwfqj\" (UniqueName: \"kubernetes.io/projected/125e80df-4f04-4843-8003-436343b5b624-kube-api-access-fwfqj\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:56:16.834459 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:16.834449    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125e80df-4f04-4843-8003-436343b5b624-plugins-config-volume\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:56:17.248653 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:17.248620    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:17.571528 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:17.571428    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:17.577353 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:17.577318    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d" event={"ID":"125e80df-4f04-4843-8003-436343b5b624","Type":"ContainerDied","Data":"2ea98c85e677a043023ab368526acc8d5d68dbd1dc13f20d3d23363f06dee988"}
Sep 10 04:56:17.577353 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:17.577353    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d"
Sep 10 04:56:17.577564 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:17.577376    2575 scope.go:117] "RemoveContainer" containerID="21a5b15c2c288caf611aa30344e7b02f0aafc011bc49fe24cb704b2d3090cc1c"
Sep 10 04:56:17.602211 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:17.602172    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d"]
Sep 10 04:56:17.606524 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:17.606502    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d778192d-56c99d84db-pll6d"]
Sep 10 04:56:17.759250 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:17.759214    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:17.862782 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:17.862707    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="125e80df-4f04-4843-8003-436343b5b624" path="/var/lib/kubelet/pods/125e80df-4f04-4843-8003-436343b5b624/volumes"
Sep 10 04:56:17.863039 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:17.863026    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed7e02f4-7f48-4b86-b7f2-398de2ab6a56" path="/var/lib/kubelet/pods/ed7e02f4-7f48-4b86-b7f2-398de2ab6a56/volumes"
Sep 10 04:56:20.300281 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:20.300245    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:20.403582 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:20.403547    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:21.520228 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:21.520194    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:21.618156 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:21.618109    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:21.763601 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:21.763566    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:21.967434 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:21.967397    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:22.046602 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:22.046566    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:44.091059 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.091022    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"]
Sep 10 04:56:44.095420 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.091429    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed7e02f4-7f48-4b86-b7f2-398de2ab6a56" containerName="revoke-keys"
Sep 10 04:56:44.095420 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.091444    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed7e02f4-7f48-4b86-b7f2-398de2ab6a56" containerName="revoke-keys"
Sep 10 04:56:44.095420 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.091454    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="125e80df-4f04-4843-8003-436343b5b624" containerName="payload-processing"
Sep 10 04:56:44.095420 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.091460    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="125e80df-4f04-4843-8003-436343b5b624" containerName="payload-processing"
Sep 10 04:56:44.095420 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.091539    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="125e80df-4f04-4843-8003-436343b5b624" containerName="payload-processing"
Sep 10 04:56:44.095420 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.091550    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed7e02f4-7f48-4b86-b7f2-398de2ab6a56" containerName="revoke-keys"
Sep 10 04:56:44.099232 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.099201    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:44.101591 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.101567    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"]
Sep 10 04:56:44.102959 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.102938    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-13af27\"/\"kube-root-ca.crt\""
Sep 10 04:56:44.103168 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.102939    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-13af27\"/\"simulator-w4-13af27-kserve-self-signed-certs\""
Sep 10 04:56:44.103285 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.102959    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-13af27\"/\"openshift-service-ca.crt\""
Sep 10 04:56:44.103352 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.103290    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-13af27\"/\"default-dockercfg-ttw9m\""
Sep 10 04:56:44.175994 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.175960    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-home\") pod \"simulator-w4-13af27-kserve-57cd7c8c65-25xtg\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") " pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:44.175994 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.175997    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-tls-certs\") pod \"simulator-w4-13af27-kserve-57cd7c8c65-25xtg\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") " pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:44.176220 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.176030    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-tmp-dir\") pod \"simulator-w4-13af27-kserve-57cd7c8c65-25xtg\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") " pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:44.176220 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.176059    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-dshm\") pod \"simulator-w4-13af27-kserve-57cd7c8c65-25xtg\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") " pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:44.176220 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.176117    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-model-cache\") pod \"simulator-w4-13af27-kserve-57cd7c8c65-25xtg\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") " pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:44.176220 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.176169    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltrbk\" (UniqueName: \"kubernetes.io/projected/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-kube-api-access-ltrbk\") pod \"simulator-w4-13af27-kserve-57cd7c8c65-25xtg\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") " pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:44.199198 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.199167    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:44.277272 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.277235    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltrbk\" (UniqueName: \"kubernetes.io/projected/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-kube-api-access-ltrbk\") pod \"simulator-w4-13af27-kserve-57cd7c8c65-25xtg\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") " pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:44.277439 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.277355    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-home\") pod \"simulator-w4-13af27-kserve-57cd7c8c65-25xtg\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") " pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:44.277439 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.277384    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-tls-certs\") pod \"simulator-w4-13af27-kserve-57cd7c8c65-25xtg\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") " pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:44.277439 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.277414    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-tmp-dir\") pod \"simulator-w4-13af27-kserve-57cd7c8c65-25xtg\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") " pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:44.277589 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.277439    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-dshm\") pod \"simulator-w4-13af27-kserve-57cd7c8c65-25xtg\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") " pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:44.277589 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.277460    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-model-cache\") pod \"simulator-w4-13af27-kserve-57cd7c8c65-25xtg\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") " pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:44.277850 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.277825    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-tmp-dir\") pod \"simulator-w4-13af27-kserve-57cd7c8c65-25xtg\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") " pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:44.277850 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.277839    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-home\") pod \"simulator-w4-13af27-kserve-57cd7c8c65-25xtg\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") " pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:44.277947 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.277870    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-model-cache\") pod \"simulator-w4-13af27-kserve-57cd7c8c65-25xtg\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") " pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:44.279769 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.279748    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-dshm\") pod \"simulator-w4-13af27-kserve-57cd7c8c65-25xtg\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") " pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:44.279938 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.279918    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-tls-certs\") pod \"simulator-w4-13af27-kserve-57cd7c8c65-25xtg\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") " pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:44.286558 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.286538    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltrbk\" (UniqueName: \"kubernetes.io/projected/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-kube-api-access-ltrbk\") pod \"simulator-w4-13af27-kserve-57cd7c8c65-25xtg\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") " pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:44.410093 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.409990    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:44.678768 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.678740    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"]
Sep 10 04:56:44.680851 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:56:44.680825    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78e74217_c203_4f2d_b38b_4ef9bbacd9a4.slice/crio-a65c77aff9cedee3d4c00bcfadfcd33d426eede8c0ffcfee8f7f20890fb38af9 WatchSource:0}: Error finding container a65c77aff9cedee3d4c00bcfadfcd33d426eede8c0ffcfee8f7f20890fb38af9: Status 404 returned error can't find the container with id a65c77aff9cedee3d4c00bcfadfcd33d426eede8c0ffcfee8f7f20890fb38af9
Sep 10 04:56:44.712976 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:44.712939    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg" event={"ID":"78e74217-c203-4f2d-b38b-4ef9bbacd9a4","Type":"ContainerStarted","Data":"a65c77aff9cedee3d4c00bcfadfcd33d426eede8c0ffcfee8f7f20890fb38af9"}
Sep 10 04:56:45.718918 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:45.718880    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg" event={"ID":"78e74217-c203-4f2d-b38b-4ef9bbacd9a4","Type":"ContainerStarted","Data":"63fd3561dced1fafb350ea3abf45159a60fa956642118ec2903a98a919be60b1"}
Sep 10 04:56:45.735999 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:45.735946    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg" podStartSLOduration=1.7359307400000001 podStartE2EDuration="1.73593074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:56:45.733508286 +0000 UTC m=+942.495277469" watchObservedRunningTime="2026-09-10 04:56:45.73593074 +0000 UTC m=+942.497699960"
Sep 10 04:56:46.055497 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:46.055405    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4df850-8mg4f"]
Sep 10 04:56:46.059059 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:46.059037    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4df850-8mg4f"
Sep 10 04:56:46.061595 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:46.061571    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 04:56:46.061715 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:46.061624    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dkf6w\""
Sep 10 04:56:46.067256 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:46.067233    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4df850-8mg4f"]
Sep 10 04:56:46.090487 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:46.090451    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b3af8a7-8117-44fa-bf8b-78c668043fa1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb4df850-8mg4f\" (UID: \"7b3af8a7-8117-44fa-bf8b-78c668043fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4df850-8mg4f"
Sep 10 04:56:46.191276 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:46.191237    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b3af8a7-8117-44fa-bf8b-78c668043fa1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb4df850-8mg4f\" (UID: \"7b3af8a7-8117-44fa-bf8b-78c668043fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4df850-8mg4f"
Sep 10 04:56:46.191844 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:46.191823    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b3af8a7-8117-44fa-bf8b-78c668043fa1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb4df850-8mg4f\" (UID: \"7b3af8a7-8117-44fa-bf8b-78c668043fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4df850-8mg4f"
Sep 10 04:56:46.371433 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:46.371395    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4df850-8mg4f"
Sep 10 04:56:46.640750 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:46.640720    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4df850-8mg4f"]
Sep 10 04:56:46.642378 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:56:46.642347    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3af8a7_8117_44fa_bf8b_78c668043fa1.slice/crio-681325798cbbea08829f6d5d94df90cd765d2d1ccbb57c2f13237af87f8435fd WatchSource:0}: Error finding container 681325798cbbea08829f6d5d94df90cd765d2d1ccbb57c2f13237af87f8435fd: Status 404 returned error can't find the container with id 681325798cbbea08829f6d5d94df90cd765d2d1ccbb57c2f13237af87f8435fd
Sep 10 04:56:46.724858 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:46.724831    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4df850-8mg4f" event={"ID":"7b3af8a7-8117-44fa-bf8b-78c668043fa1","Type":"ContainerStarted","Data":"681325798cbbea08829f6d5d94df90cd765d2d1ccbb57c2f13237af87f8435fd"}
Sep 10 04:56:47.730767 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:47.730732    2575 generic.go:356] "Generic (PLEG): container finished" podID="7b3af8a7-8117-44fa-bf8b-78c668043fa1" containerID="aa9151d1f3cb6ff8e63445fd247370628a38e968269ca88f1c0de3e9a98516bc" exitCode=0
Sep 10 04:56:47.731182 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:47.730790    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4df850-8mg4f" event={"ID":"7b3af8a7-8117-44fa-bf8b-78c668043fa1","Type":"ContainerDied","Data":"aa9151d1f3cb6ff8e63445fd247370628a38e968269ca88f1c0de3e9a98516bc"}
Sep 10 04:56:48.861356 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:48.861331    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4df850-8mg4f"
Sep 10 04:56:48.910518 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:48.910477    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b3af8a7-8117-44fa-bf8b-78c668043fa1-maas-api-service-ca\") pod \"7b3af8a7-8117-44fa-bf8b-78c668043fa1\" (UID: \"7b3af8a7-8117-44fa-bf8b-78c668043fa1\") "
Sep 10 04:56:48.910797 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:48.910773    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b3af8a7-8117-44fa-bf8b-78c668043fa1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7b3af8a7-8117-44fa-bf8b-78c668043fa1" (UID: "7b3af8a7-8117-44fa-bf8b-78c668043fa1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 04:56:49.011471 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:49.011382    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b3af8a7-8117-44fa-bf8b-78c668043fa1-maas-api-service-ca\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:56:49.749447 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:49.749418    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4df850-8mg4f"
Sep 10 04:56:49.749619 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:49.749412    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4df850-8mg4f" event={"ID":"7b3af8a7-8117-44fa-bf8b-78c668043fa1","Type":"ContainerDied","Data":"681325798cbbea08829f6d5d94df90cd765d2d1ccbb57c2f13237af87f8435fd"}
Sep 10 04:56:49.749619 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:49.749577    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="681325798cbbea08829f6d5d94df90cd765d2d1ccbb57c2f13237af87f8435fd"
Sep 10 04:56:52.105809 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:52.105772    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:52.168987 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:52.168951    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:52.270248 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:52.270206    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:52.287856 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:52.287826    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4df850-8mg4f"]
Sep 10 04:56:52.290595 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:52.290570    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4df850-8mg4f"]
Sep 10 04:56:52.335960 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:52.335924    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:53.482151 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:53.482114    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:53.863121 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:53.863022    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b3af8a7-8117-44fa-bf8b-78c668043fa1" path="/var/lib/kubelet/pods/7b3af8a7-8117-44fa-bf8b-78c668043fa1/volumes"
Sep 10 04:56:53.918945 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:53.918908    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:54.226869 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:54.226840    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:54.411078 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:54.411043    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:54.411240 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:54.411104    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:54.423912 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:54.423881    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:54.434585 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:54.434552    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:54.784618 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:54.784589    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 04:56:55.181524 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:55.181445    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:57.246649 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:57.246612    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:57.345060 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:57.345020    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:58.425055 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:58.425016    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:58.525967 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:58.525929    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:58.656086 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:58.656033    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:58.969576 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:58.969540    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:59.071411 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.071373    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:59.379318 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.379234    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"]
Sep 10 04:56:59.379641 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.379624    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b3af8a7-8117-44fa-bf8b-78c668043fa1" containerName="revoke-keys"
Sep 10 04:56:59.379641 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.379641    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b3af8a7-8117-44fa-bf8b-78c668043fa1" containerName="revoke-keys"
Sep 10 04:56:59.379728 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.379701    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b3af8a7-8117-44fa-bf8b-78c668043fa1" containerName="revoke-keys"
Sep 10 04:56:59.382986 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.382969    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:56:59.386514 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.386489    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-53c734\"/\"default-dockercfg-24n7f\""
Sep 10 04:56:59.386645 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.386522    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-53c734\"/\"openshift-service-ca.crt\""
Sep 10 04:56:59.386645 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.386539    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-53c734\"/\"autores-model-53c734-kserve-self-signed-certs\""
Sep 10 04:56:59.386645 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.386489    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-53c734\"/\"kube-root-ca.crt\""
Sep 10 04:56:59.390661 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.390635    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"]
Sep 10 04:56:59.490054 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.490013    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-model-cache\") pod \"autores-model-53c734-kserve-7ffd88cc6c-rml69\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") " pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:56:59.490522 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.490095    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d39a99-d3af-4045-9e30-b3599775ab15-tls-certs\") pod \"autores-model-53c734-kserve-7ffd88cc6c-rml69\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") " pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:56:59.490522 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.490144    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-dshm\") pod \"autores-model-53c734-kserve-7ffd88cc6c-rml69\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") " pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:56:59.490522 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.490181    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzvd7\" (UniqueName: \"kubernetes.io/projected/00d39a99-d3af-4045-9e30-b3599775ab15-kube-api-access-rzvd7\") pod \"autores-model-53c734-kserve-7ffd88cc6c-rml69\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") " pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:56:59.490522 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.490276    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-home\") pod \"autores-model-53c734-kserve-7ffd88cc6c-rml69\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") " pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:56:59.490522 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.490298    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-tmp-dir\") pod \"autores-model-53c734-kserve-7ffd88cc6c-rml69\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") " pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:56:59.591688 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.591653    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-home\") pod \"autores-model-53c734-kserve-7ffd88cc6c-rml69\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") " pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:56:59.591688 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.591690    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-tmp-dir\") pod \"autores-model-53c734-kserve-7ffd88cc6c-rml69\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") " pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:56:59.591908 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.591820    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-model-cache\") pod \"autores-model-53c734-kserve-7ffd88cc6c-rml69\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") " pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:56:59.591908 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.591884    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d39a99-d3af-4045-9e30-b3599775ab15-tls-certs\") pod \"autores-model-53c734-kserve-7ffd88cc6c-rml69\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") " pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:56:59.591990 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.591948    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-dshm\") pod \"autores-model-53c734-kserve-7ffd88cc6c-rml69\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") " pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:56:59.592044 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.591997    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzvd7\" (UniqueName: \"kubernetes.io/projected/00d39a99-d3af-4045-9e30-b3599775ab15-kube-api-access-rzvd7\") pod \"autores-model-53c734-kserve-7ffd88cc6c-rml69\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") " pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:56:59.592151 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.592129    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-home\") pod \"autores-model-53c734-kserve-7ffd88cc6c-rml69\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") " pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:56:59.592213 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.592152    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-tmp-dir\") pod \"autores-model-53c734-kserve-7ffd88cc6c-rml69\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") " pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:56:59.592270 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.592245    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-model-cache\") pod \"autores-model-53c734-kserve-7ffd88cc6c-rml69\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") " pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:56:59.594224 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.594200    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-dshm\") pod \"autores-model-53c734-kserve-7ffd88cc6c-rml69\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") " pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:56:59.594760 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.594744    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d39a99-d3af-4045-9e30-b3599775ab15-tls-certs\") pod \"autores-model-53c734-kserve-7ffd88cc6c-rml69\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") " pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:56:59.599458 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.599438    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzvd7\" (UniqueName: \"kubernetes.io/projected/00d39a99-d3af-4045-9e30-b3599775ab15-kube-api-access-rzvd7\") pod \"autores-model-53c734-kserve-7ffd88cc6c-rml69\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") " pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:56:59.692876 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.692784    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:56:59.791023 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.790933    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:56:59.962856 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:56:59.962825    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"]
Sep 10 04:56:59.964507 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:56:59.964483    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00d39a99_d3af_4045_9e30_b3599775ab15.slice/crio-a817556a9dea30a1c07a99f3ccb90c0243ebffc0944b7bea042b3a67810b87e7 WatchSource:0}: Error finding container a817556a9dea30a1c07a99f3ccb90c0243ebffc0944b7bea042b3a67810b87e7: Status 404 returned error can't find the container with id a817556a9dea30a1c07a99f3ccb90c0243ebffc0944b7bea042b3a67810b87e7
Sep 10 04:57:00.803545 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:00.803503    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69" event={"ID":"00d39a99-d3af-4045-9e30-b3599775ab15","Type":"ContainerStarted","Data":"4e42b23ad6b29e0432dca095aa512346f6ec6ae257375a031fce6c8374caacc6"}
Sep 10 04:57:00.803921 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:00.803555    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69" event={"ID":"00d39a99-d3af-4045-9e30-b3599775ab15","Type":"ContainerStarted","Data":"a817556a9dea30a1c07a99f3ccb90c0243ebffc0944b7bea042b3a67810b87e7"}
Sep 10 04:57:01.829730 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:01.829697    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:09.692895 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:09.692851    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:57:09.693400 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:09.692909    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:57:09.705625 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:09.705600    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:57:09.722917 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:09.722868    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69" podStartSLOduration=10.722852816 podStartE2EDuration="10.722852816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:57:00.819931444 +0000 UTC m=+957.581700629" watchObservedRunningTime="2026-09-10 04:57:09.722852816 +0000 UTC m=+966.484621997"
Sep 10 04:57:09.856292 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:09.856260    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:57:19.417722 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:19.417681    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:22.108966 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:22.108931    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ead8c2c2-p7zjf"]
Sep 10 04:57:22.113817 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:22.113799    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ead8c2c2-p7zjf"
Sep 10 04:57:22.117494 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:22.117350    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dkf6w\""
Sep 10 04:57:22.117863 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:22.117838    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 04:57:22.118939 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:22.118917    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ead8c2c2-p7zjf"]
Sep 10 04:57:22.207141 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:22.207101    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a5e256-7aee-43bc-ad9b-f9189d1a0281-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ead8c2c2-p7zjf\" (UID: \"33a5e256-7aee-43bc-ad9b-f9189d1a0281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ead8c2c2-p7zjf"
Sep 10 04:57:22.308324 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:22.308280    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a5e256-7aee-43bc-ad9b-f9189d1a0281-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ead8c2c2-p7zjf\" (UID: \"33a5e256-7aee-43bc-ad9b-f9189d1a0281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ead8c2c2-p7zjf"
Sep 10 04:57:22.308899 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:22.308880    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a5e256-7aee-43bc-ad9b-f9189d1a0281-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ead8c2c2-p7zjf\" (UID: \"33a5e256-7aee-43bc-ad9b-f9189d1a0281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ead8c2c2-p7zjf"
Sep 10 04:57:22.426689 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:22.426609    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ead8c2c2-p7zjf"
Sep 10 04:57:22.697121 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:22.697096    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ead8c2c2-p7zjf"]
Sep 10 04:57:22.698627 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:57:22.698592    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a5e256_7aee_43bc_ad9b_f9189d1a0281.slice/crio-aa554743ed26a6ea76aabb700e3a8696b7b6e6cd321a98023d31439c5fe07eef WatchSource:0}: Error finding container aa554743ed26a6ea76aabb700e3a8696b7b6e6cd321a98023d31439c5fe07eef: Status 404 returned error can't find the container with id aa554743ed26a6ea76aabb700e3a8696b7b6e6cd321a98023d31439c5fe07eef
Sep 10 04:57:22.913606 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:22.913573    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ead8c2c2-p7zjf" event={"ID":"33a5e256-7aee-43bc-ad9b-f9189d1a0281","Type":"ContainerStarted","Data":"aa554743ed26a6ea76aabb700e3a8696b7b6e6cd321a98023d31439c5fe07eef"}
Sep 10 04:57:23.921195 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:23.921099    2575 generic.go:356] "Generic (PLEG): container finished" podID="33a5e256-7aee-43bc-ad9b-f9189d1a0281" containerID="e819599c4fe63f742ea85741b46f1c4eee4d878d4cf5ec09bb0834d0095cd655" exitCode=0
Sep 10 04:57:23.921195 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:23.921178    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ead8c2c2-p7zjf" event={"ID":"33a5e256-7aee-43bc-ad9b-f9189d1a0281","Type":"ContainerDied","Data":"e819599c4fe63f742ea85741b46f1c4eee4d878d4cf5ec09bb0834d0095cd655"}
Sep 10 04:57:25.057747 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:25.057719    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ead8c2c2-p7zjf"
Sep 10 04:57:25.136274 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:25.136249    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a5e256-7aee-43bc-ad9b-f9189d1a0281-maas-api-service-ca\") pod \"33a5e256-7aee-43bc-ad9b-f9189d1a0281\" (UID: \"33a5e256-7aee-43bc-ad9b-f9189d1a0281\") "
Sep 10 04:57:25.136610 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:25.136589    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a5e256-7aee-43bc-ad9b-f9189d1a0281-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "33a5e256-7aee-43bc-ad9b-f9189d1a0281" (UID: "33a5e256-7aee-43bc-ad9b-f9189d1a0281"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 04:57:25.236850 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:25.236821    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a5e256-7aee-43bc-ad9b-f9189d1a0281-maas-api-service-ca\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:57:25.931865 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:25.931829    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ead8c2c2-p7zjf" event={"ID":"33a5e256-7aee-43bc-ad9b-f9189d1a0281","Type":"ContainerDied","Data":"aa554743ed26a6ea76aabb700e3a8696b7b6e6cd321a98023d31439c5fe07eef"}
Sep 10 04:57:25.931865 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:25.931849    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ead8c2c2-p7zjf"
Sep 10 04:57:25.932112 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:25.931875    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa554743ed26a6ea76aabb700e3a8696b7b6e6cd321a98023d31439c5fe07eef"
Sep 10 04:57:28.882611 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:28.882566    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:28.954262 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:28.954224    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:29.014381 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:29.014341    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:29.068952 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:29.068916    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ead8c2c2-p7zjf"]
Sep 10 04:57:29.070664 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:29.070641    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ead8c2c2-p7zjf"]
Sep 10 04:57:29.072982 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:29.072923    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:29.126713 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:29.126680    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:29.219939 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:29.219904    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:29.862793 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:29.862761    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33a5e256-7aee-43bc-ad9b-f9189d1a0281" path="/var/lib/kubelet/pods/33a5e256-7aee-43bc-ad9b-f9189d1a0281/volumes"
Sep 10 04:57:29.916223 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:29.916143    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:30.007934 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:30.007890    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:30.107350 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:30.107313    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:31.572683 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:31.572647    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:32.936285 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:32.936252    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:35.015104 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:35.015056    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-53c734-62vl9"]
Sep 10 04:57:35.015474 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:35.015449    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33a5e256-7aee-43bc-ad9b-f9189d1a0281" containerName="revoke-keys"
Sep 10 04:57:35.015474 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:35.015463    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="33a5e256-7aee-43bc-ad9b-f9189d1a0281" containerName="revoke-keys"
Sep 10 04:57:35.015548 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:35.015535    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="33a5e256-7aee-43bc-ad9b-f9189d1a0281" containerName="revoke-keys"
Sep 10 04:57:35.018462 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:35.018446    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-53c734-62vl9"
Sep 10 04:57:35.020763 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:35.020737    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 04:57:35.020763 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:35.020737    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dkf6w\""
Sep 10 04:57:35.024781 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:35.024757    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-53c734-62vl9"]
Sep 10 04:57:35.034934 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:35.034910    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e52e97-91c7-4261-b717-c52961635ae7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-53c734-62vl9\" (UID: \"b1e52e97-91c7-4261-b717-c52961635ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-53c734-62vl9"
Sep 10 04:57:35.135949 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:35.135915    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e52e97-91c7-4261-b717-c52961635ae7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-53c734-62vl9\" (UID: \"b1e52e97-91c7-4261-b717-c52961635ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-53c734-62vl9"
Sep 10 04:57:35.136576 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:35.136553    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e52e97-91c7-4261-b717-c52961635ae7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-53c734-62vl9\" (UID: \"b1e52e97-91c7-4261-b717-c52961635ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-53c734-62vl9"
Sep 10 04:57:35.236421 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:35.236383    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:35.330143 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:35.330039    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-53c734-62vl9"
Sep 10 04:57:35.597311 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:35.597286    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-53c734-62vl9"]
Sep 10 04:57:35.599848 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:57:35.599820    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e52e97_91c7_4261_b717_c52961635ae7.slice/crio-404d956afe3f2aafa92672bf42a9955aa5afc1ca55ba90403b296b3628ec4ad4 WatchSource:0}: Error finding container 404d956afe3f2aafa92672bf42a9955aa5afc1ca55ba90403b296b3628ec4ad4: Status 404 returned error can't find the container with id 404d956afe3f2aafa92672bf42a9955aa5afc1ca55ba90403b296b3628ec4ad4
Sep 10 04:57:35.984363 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:35.984318    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-53c734-62vl9" event={"ID":"b1e52e97-91c7-4261-b717-c52961635ae7","Type":"ContainerStarted","Data":"e39d621bb4e49ffe6d944a373ec22370a52a4000b07cbbb5f5575819763ca919"}
Sep 10 04:57:35.984535 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:35.984372    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-53c734-62vl9" event={"ID":"b1e52e97-91c7-4261-b717-c52961635ae7","Type":"ContainerStarted","Data":"404d956afe3f2aafa92672bf42a9955aa5afc1ca55ba90403b296b3628ec4ad4"}
Sep 10 04:57:35.999973 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:35.999923    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-53c734-62vl9" podStartSLOduration=0.999907868 podStartE2EDuration="999.907868ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:57:35.997946409 +0000 UTC m=+992.759715591" watchObservedRunningTime="2026-09-10 04:57:35.999907868 +0000 UTC m=+992.761677049"
Sep 10 04:57:36.990320 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:36.990286    2575 generic.go:356] "Generic (PLEG): container finished" podID="b1e52e97-91c7-4261-b717-c52961635ae7" containerID="e39d621bb4e49ffe6d944a373ec22370a52a4000b07cbbb5f5575819763ca919" exitCode=0
Sep 10 04:57:36.990709 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:36.990348    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-53c734-62vl9" event={"ID":"b1e52e97-91c7-4261-b717-c52961635ae7","Type":"ContainerDied","Data":"e39d621bb4e49ffe6d944a373ec22370a52a4000b07cbbb5f5575819763ca919"}
Sep 10 04:57:37.267810 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:37.267724    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:37.370243 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:37.370211    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:37.468676 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:37.468637    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:37.568931 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:37.568838    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:37.950602 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:37.950564    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:38.127381 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:38.127341    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-53c734-62vl9"
Sep 10 04:57:38.163220 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:38.163186    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e52e97-91c7-4261-b717-c52961635ae7-maas-api-service-ca\") pod \"b1e52e97-91c7-4261-b717-c52961635ae7\" (UID: \"b1e52e97-91c7-4261-b717-c52961635ae7\") "
Sep 10 04:57:38.163614 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:38.163585    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1e52e97-91c7-4261-b717-c52961635ae7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b1e52e97-91c7-4261-b717-c52961635ae7" (UID: "b1e52e97-91c7-4261-b717-c52961635ae7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 04:57:38.264546 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:38.264459    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e52e97-91c7-4261-b717-c52961635ae7-maas-api-service-ca\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:57:39.001812 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:39.001772    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-53c734-62vl9" event={"ID":"b1e52e97-91c7-4261-b717-c52961635ae7","Type":"ContainerDied","Data":"404d956afe3f2aafa92672bf42a9955aa5afc1ca55ba90403b296b3628ec4ad4"}
Sep 10 04:57:39.001983 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:39.001832    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="404d956afe3f2aafa92672bf42a9955aa5afc1ca55ba90403b296b3628ec4ad4"
Sep 10 04:57:39.001983 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:39.001783    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-53c734-62vl9"
Sep 10 04:57:45.551553 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:45.551509    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:45.620464 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:45.620417    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:45.679828 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:45.679786    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:45.742416 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:45.742382    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-53c734-62vl9"]
Sep 10 04:57:45.744243 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:45.744221    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-53c734-62vl9"]
Sep 10 04:57:45.747364 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:45.747337    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:45.800346 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:45.800319    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:45.862702 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:45.862618    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1e52e97-91c7-4261-b717-c52961635ae7" path="/var/lib/kubelet/pods/b1e52e97-91c7-4261-b717-c52961635ae7/volumes"
Sep 10 04:57:46.115605 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:46.115525    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:57:51.098572 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:51.098527    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"]
Sep 10 04:57:51.139760 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:57:51.139725    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:03.898153 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:03.898108    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:06.118161 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:06.118118    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69" podUID="00d39a99-d3af-4045-9e30-b3599775ab15" containerName="main" containerID="cri-o://4e42b23ad6b29e0432dca095aa512346f6ec6ae257375a031fce6c8374caacc6" gracePeriod=45
Sep 10 04:58:06.373997 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:06.373936    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:58:06.527234 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:06.527201    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzvd7\" (UniqueName: \"kubernetes.io/projected/00d39a99-d3af-4045-9e30-b3599775ab15-kube-api-access-rzvd7\") pod \"00d39a99-d3af-4045-9e30-b3599775ab15\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") "
Sep 10 04:58:06.527234 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:06.527238    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-home\") pod \"00d39a99-d3af-4045-9e30-b3599775ab15\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") "
Sep 10 04:58:06.527466 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:06.527300    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-model-cache\") pod \"00d39a99-d3af-4045-9e30-b3599775ab15\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") "
Sep 10 04:58:06.527466 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:06.527318    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-tmp-dir\") pod \"00d39a99-d3af-4045-9e30-b3599775ab15\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") "
Sep 10 04:58:06.527466 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:06.527361    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d39a99-d3af-4045-9e30-b3599775ab15-tls-certs\") pod \"00d39a99-d3af-4045-9e30-b3599775ab15\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") "
Sep 10 04:58:06.527466 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:06.527396    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-dshm\") pod \"00d39a99-d3af-4045-9e30-b3599775ab15\" (UID: \"00d39a99-d3af-4045-9e30-b3599775ab15\") "
Sep 10 04:58:06.527666 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:06.527550    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-model-cache" (OuterVolumeSpecName: "model-cache") pod "00d39a99-d3af-4045-9e30-b3599775ab15" (UID: "00d39a99-d3af-4045-9e30-b3599775ab15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 04:58:06.527666 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:06.527587    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-home" (OuterVolumeSpecName: "home") pod "00d39a99-d3af-4045-9e30-b3599775ab15" (UID: "00d39a99-d3af-4045-9e30-b3599775ab15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 04:58:06.527666 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:06.527595    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "00d39a99-d3af-4045-9e30-b3599775ab15" (UID: "00d39a99-d3af-4045-9e30-b3599775ab15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 04:58:06.527773 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:06.527673    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-home\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:58:06.527773 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:06.527687    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-model-cache\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:58:06.527773 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:06.527697    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-tmp-dir\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:58:06.529509 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:06.529478    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-dshm" (OuterVolumeSpecName: "dshm") pod "00d39a99-d3af-4045-9e30-b3599775ab15" (UID: "00d39a99-d3af-4045-9e30-b3599775ab15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 04:58:06.529630 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:06.529515    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00d39a99-d3af-4045-9e30-b3599775ab15-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "00d39a99-d3af-4045-9e30-b3599775ab15" (UID: "00d39a99-d3af-4045-9e30-b3599775ab15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 04:58:06.529630 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:06.529523    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d39a99-d3af-4045-9e30-b3599775ab15-kube-api-access-rzvd7" (OuterVolumeSpecName: "kube-api-access-rzvd7") pod "00d39a99-d3af-4045-9e30-b3599775ab15" (UID: "00d39a99-d3af-4045-9e30-b3599775ab15"). InnerVolumeSpecName "kube-api-access-rzvd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 04:58:06.628527 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:06.628451    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzvd7\" (UniqueName: \"kubernetes.io/projected/00d39a99-d3af-4045-9e30-b3599775ab15-kube-api-access-rzvd7\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:58:06.628527 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:06.628488    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d39a99-d3af-4045-9e30-b3599775ab15-tls-certs\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:58:06.628527 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:06.628499    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00d39a99-d3af-4045-9e30-b3599775ab15-dshm\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:58:07.147775 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:07.147737    2575 generic.go:356] "Generic (PLEG): container finished" podID="00d39a99-d3af-4045-9e30-b3599775ab15" containerID="4e42b23ad6b29e0432dca095aa512346f6ec6ae257375a031fce6c8374caacc6" exitCode=0
Sep 10 04:58:07.148179 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:07.147783    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"
Sep 10 04:58:07.148179 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:07.147824    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69" event={"ID":"00d39a99-d3af-4045-9e30-b3599775ab15","Type":"ContainerDied","Data":"4e42b23ad6b29e0432dca095aa512346f6ec6ae257375a031fce6c8374caacc6"}
Sep 10 04:58:07.148179 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:07.147874    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69" event={"ID":"00d39a99-d3af-4045-9e30-b3599775ab15","Type":"ContainerDied","Data":"a817556a9dea30a1c07a99f3ccb90c0243ebffc0944b7bea042b3a67810b87e7"}
Sep 10 04:58:07.148179 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:07.147894    2575 scope.go:117] "RemoveContainer" containerID="4e42b23ad6b29e0432dca095aa512346f6ec6ae257375a031fce6c8374caacc6"
Sep 10 04:58:07.157288 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:07.157272    2575 scope.go:117] "RemoveContainer" containerID="4e42b23ad6b29e0432dca095aa512346f6ec6ae257375a031fce6c8374caacc6"
Sep 10 04:58:07.157545 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:58:07.157528    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e42b23ad6b29e0432dca095aa512346f6ec6ae257375a031fce6c8374caacc6\": container with ID starting with 4e42b23ad6b29e0432dca095aa512346f6ec6ae257375a031fce6c8374caacc6 not found: ID does not exist" containerID="4e42b23ad6b29e0432dca095aa512346f6ec6ae257375a031fce6c8374caacc6"
Sep 10 04:58:07.157593 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:07.157552    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e42b23ad6b29e0432dca095aa512346f6ec6ae257375a031fce6c8374caacc6"} err="failed to get container status \"4e42b23ad6b29e0432dca095aa512346f6ec6ae257375a031fce6c8374caacc6\": rpc error: code = NotFound desc = could not find container \"4e42b23ad6b29e0432dca095aa512346f6ec6ae257375a031fce6c8374caacc6\": container with ID starting with 4e42b23ad6b29e0432dca095aa512346f6ec6ae257375a031fce6c8374caacc6 not found: ID does not exist"
Sep 10 04:58:07.175450 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:07.175410    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"]
Sep 10 04:58:07.179264 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:07.179233    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-53c734/autores-model-53c734-kserve-7ffd88cc6c-rml69"]
Sep 10 04:58:07.863660 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:07.863624    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00d39a99-d3af-4045-9e30-b3599775ab15" path="/var/lib/kubelet/pods/00d39a99-d3af-4045-9e30-b3599775ab15/volumes"
Sep 10 04:58:08.975373 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:08.975341    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:09.708092 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:09.708030    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:09.826904 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:09.826866    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:09.962239 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:09.962149    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:25.789708 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:25.789673    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:27.919986 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:27.919948    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:28.181109 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:28.181006    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:28.642888 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:28.642852    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:28.955043 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:28.955009    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:29.319387 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:29.319275    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:31.552769 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:31.552733    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:31.663610 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:31.663573    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:31.753193 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:31.753159    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:32.454958 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:32.454900    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:32.550436 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:32.550405    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:33.628893 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:33.628840    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:33.729303 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:33.729268    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:33.778102 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:33.778041    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:34.092832 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:34.092794    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:34.194479 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:34.194441    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:42.285331 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:42.285291    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:43.134593 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:43.134538    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:43.232228 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:43.232189    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:48.771484 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:48.771441    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:48.839365 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:48.839318    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:48.915583 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:48.915553    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:49.013626 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:49.013592    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:49.115114 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:49.115014    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:49.808575 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:49.808538    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:49.902702 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:49.902664    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:50.007995 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:50.007958    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:51.522116 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:51.522082    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:57.254948 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:57.254840    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:57.356376 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:57.356336    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:57.455827 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:57.455753    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:58:57.962639 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:58:57.962602    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:03.636678 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:03.636637    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:03.734771 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:03.734732    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:03.878492 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:03.878449    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:04.190694 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:04.190587    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:04.481198 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:04.481151    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:09.796419 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:09.796373    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:09.890936 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:09.890884    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:11.126464 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:11.126425    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:11.182134 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:11.182097    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:11.282531 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:11.282493    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:11.581195 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:11.581162    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:11.674181 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:11.674144    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:17.980991 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:17.980956    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"]
Sep 10 04:59:17.981381 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:17.981342    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d39a99-d3af-4045-9e30-b3599775ab15" containerName="main"
Sep 10 04:59:17.981381 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:17.981354    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d39a99-d3af-4045-9e30-b3599775ab15" containerName="main"
Sep 10 04:59:17.981381 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:17.981363    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e52e97-91c7-4261-b717-c52961635ae7" containerName="revoke-keys"
Sep 10 04:59:17.981381 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:17.981370    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e52e97-91c7-4261-b717-c52961635ae7" containerName="revoke-keys"
Sep 10 04:59:17.981520 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:17.981428    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1e52e97-91c7-4261-b717-c52961635ae7" containerName="revoke-keys"
Sep 10 04:59:17.981520 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:17.981440    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="00d39a99-d3af-4045-9e30-b3599775ab15" containerName="main"
Sep 10 04:59:17.984924 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:17.984907    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:17.988764 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:17.988742    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-ea9ba2\"/\"explicit-model-ea9ba2-kserve-self-signed-certs\""
Sep 10 04:59:17.988862 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:17.988741    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-ea9ba2\"/\"kube-root-ca.crt\""
Sep 10 04:59:17.988862 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:17.988744    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-ea9ba2\"/\"openshift-service-ca.crt\""
Sep 10 04:59:17.988862 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:17.988843    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-ea9ba2\"/\"default-dockercfg-8pgv4\""
Sep 10 04:59:17.992788 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:17.992772    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"]
Sep 10 04:59:18.059460 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:18.059426    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-tmp-dir\") pod \"explicit-model-ea9ba2-kserve-5887b7f897-b49xg\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") " pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:18.059460 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:18.059462    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-home\") pod \"explicit-model-ea9ba2-kserve-5887b7f897-b49xg\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") " pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:18.059637 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:18.059481    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-dshm\") pod \"explicit-model-ea9ba2-kserve-5887b7f897-b49xg\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") " pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:18.059637 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:18.059561    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjq2z\" (UniqueName: \"kubernetes.io/projected/1bcdcb4e-8de9-4158-b164-30df2b15924a-kube-api-access-zjq2z\") pod \"explicit-model-ea9ba2-kserve-5887b7f897-b49xg\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") " pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:18.059637 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:18.059617    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-model-cache\") pod \"explicit-model-ea9ba2-kserve-5887b7f897-b49xg\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") " pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:18.059737 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:18.059658    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bcdcb4e-8de9-4158-b164-30df2b15924a-tls-certs\") pod \"explicit-model-ea9ba2-kserve-5887b7f897-b49xg\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") " pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:18.161145 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:18.161097    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-model-cache\") pod \"explicit-model-ea9ba2-kserve-5887b7f897-b49xg\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") " pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:18.161318 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:18.161167    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bcdcb4e-8de9-4158-b164-30df2b15924a-tls-certs\") pod \"explicit-model-ea9ba2-kserve-5887b7f897-b49xg\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") " pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:18.161318 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:18.161223    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-tmp-dir\") pod \"explicit-model-ea9ba2-kserve-5887b7f897-b49xg\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") " pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:18.161318 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:18.161250    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-home\") pod \"explicit-model-ea9ba2-kserve-5887b7f897-b49xg\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") " pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:18.161318 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:18.161280    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-dshm\") pod \"explicit-model-ea9ba2-kserve-5887b7f897-b49xg\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") " pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:18.161494 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:18.161339    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjq2z\" (UniqueName: \"kubernetes.io/projected/1bcdcb4e-8de9-4158-b164-30df2b15924a-kube-api-access-zjq2z\") pod \"explicit-model-ea9ba2-kserve-5887b7f897-b49xg\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") " pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:18.161550 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:18.161516    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-model-cache\") pod \"explicit-model-ea9ba2-kserve-5887b7f897-b49xg\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") " pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:18.161630 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:18.161598    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-home\") pod \"explicit-model-ea9ba2-kserve-5887b7f897-b49xg\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") " pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:18.161630 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:18.161625    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-tmp-dir\") pod \"explicit-model-ea9ba2-kserve-5887b7f897-b49xg\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") " pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:18.163545 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:18.163523    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-dshm\") pod \"explicit-model-ea9ba2-kserve-5887b7f897-b49xg\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") " pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:18.163729 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:18.163712    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bcdcb4e-8de9-4158-b164-30df2b15924a-tls-certs\") pod \"explicit-model-ea9ba2-kserve-5887b7f897-b49xg\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") " pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:18.168907 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:18.168888    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjq2z\" (UniqueName: \"kubernetes.io/projected/1bcdcb4e-8de9-4158-b164-30df2b15924a-kube-api-access-zjq2z\") pod \"explicit-model-ea9ba2-kserve-5887b7f897-b49xg\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") " pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:18.292827 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:18.292738    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:18.294325 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:18.294297    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:18.568234 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:18.568207    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"]
Sep 10 04:59:18.570599 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:59:18.570572    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bcdcb4e_8de9_4158_b164_30df2b15924a.slice/crio-e1b07c9ca91eb8cea6b2404083dbf3a02e5709c7953f985b86094a76b8300bfd WatchSource:0}: Error finding container e1b07c9ca91eb8cea6b2404083dbf3a02e5709c7953f985b86094a76b8300bfd: Status 404 returned error can't find the container with id e1b07c9ca91eb8cea6b2404083dbf3a02e5709c7953f985b86094a76b8300bfd
Sep 10 04:59:19.504776 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:19.504742    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg" event={"ID":"1bcdcb4e-8de9-4158-b164-30df2b15924a","Type":"ContainerStarted","Data":"870ddc054cdb87883e5bf9c24f8b0f4c3d2adc368228342154b50f0e5ce73e6a"}
Sep 10 04:59:19.505219 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:19.504785    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg" event={"ID":"1bcdcb4e-8de9-4158-b164-30df2b15924a","Type":"ContainerStarted","Data":"e1b07c9ca91eb8cea6b2404083dbf3a02e5709c7953f985b86094a76b8300bfd"}
Sep 10 04:59:19.522714 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:19.522664    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg" podStartSLOduration=2.522649078 podStartE2EDuration="2.522649078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:59:19.520730426 +0000 UTC m=+1096.282499609" watchObservedRunningTime="2026-09-10 04:59:19.522649078 +0000 UTC m=+1096.284418260"
Sep 10 04:59:24.231593 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:24.231558    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"]
Sep 10 04:59:24.250854 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:24.250822    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:24.387605 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:24.387572    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea9ba2-r6n2l"]
Sep 10 04:59:24.391042 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:24.391025    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea9ba2-r6n2l"
Sep 10 04:59:24.393546 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:24.393524    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 04:59:24.393546 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:24.393541    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dkf6w\""
Sep 10 04:59:24.397587 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:24.397565    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea9ba2-r6n2l"]
Sep 10 04:59:24.517408 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:24.517318    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daf8e8c1-ad28-4a0d-9d2c-f619176037c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-ea9ba2-r6n2l\" (UID: \"daf8e8c1-ad28-4a0d-9d2c-f619176037c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea9ba2-r6n2l"
Sep 10 04:59:24.618288 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:24.618253    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daf8e8c1-ad28-4a0d-9d2c-f619176037c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-ea9ba2-r6n2l\" (UID: \"daf8e8c1-ad28-4a0d-9d2c-f619176037c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea9ba2-r6n2l"
Sep 10 04:59:24.618849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:24.618829    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daf8e8c1-ad28-4a0d-9d2c-f619176037c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-ea9ba2-r6n2l\" (UID: \"daf8e8c1-ad28-4a0d-9d2c-f619176037c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea9ba2-r6n2l"
Sep 10 04:59:24.630222 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:24.630185    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:24.703965 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:24.703929    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea9ba2-r6n2l"
Sep 10 04:59:24.978390 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:24.978354    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea9ba2-r6n2l"]
Sep 10 04:59:24.981522 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:59:24.981491    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf8e8c1_ad28_4a0d_9d2c_f619176037c7.slice/crio-6242931a478f4be21cbe5dc5aeae99f41db6bcb66ee8f194e49d06e3076cd0f8 WatchSource:0}: Error finding container 6242931a478f4be21cbe5dc5aeae99f41db6bcb66ee8f194e49d06e3076cd0f8: Status 404 returned error can't find the container with id 6242931a478f4be21cbe5dc5aeae99f41db6bcb66ee8f194e49d06e3076cd0f8
Sep 10 04:59:25.536189 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:25.536101    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea9ba2-r6n2l" event={"ID":"daf8e8c1-ad28-4a0d-9d2c-f619176037c7","Type":"ContainerStarted","Data":"15d56d8d2958cbe54654af08df863556dfe079b234586a4e103c42fdbfacd830"}
Sep 10 04:59:25.536189 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:25.536173    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea9ba2-r6n2l" event={"ID":"daf8e8c1-ad28-4a0d-9d2c-f619176037c7","Type":"ContainerStarted","Data":"6242931a478f4be21cbe5dc5aeae99f41db6bcb66ee8f194e49d06e3076cd0f8"}
Sep 10 04:59:25.553912 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:25.553855    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea9ba2-r6n2l" podStartSLOduration=1.553841335 podStartE2EDuration="1.553841335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:59:25.551544994 +0000 UTC m=+1102.313314176" watchObservedRunningTime="2026-09-10 04:59:25.553841335 +0000 UTC m=+1102.315610517"
Sep 10 04:59:26.542701 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:26.542668    2575 generic.go:356] "Generic (PLEG): container finished" podID="daf8e8c1-ad28-4a0d-9d2c-f619176037c7" containerID="15d56d8d2958cbe54654af08df863556dfe079b234586a4e103c42fdbfacd830" exitCode=0
Sep 10 04:59:26.543108 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:26.542759    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea9ba2-r6n2l" event={"ID":"daf8e8c1-ad28-4a0d-9d2c-f619176037c7","Type":"ContainerDied","Data":"15d56d8d2958cbe54654af08df863556dfe079b234586a4e103c42fdbfacd830"}
Sep 10 04:59:27.687037 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:27.687012    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea9ba2-r6n2l"
Sep 10 04:59:27.749673 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:27.749640    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daf8e8c1-ad28-4a0d-9d2c-f619176037c7-maas-api-service-ca\") pod \"daf8e8c1-ad28-4a0d-9d2c-f619176037c7\" (UID: \"daf8e8c1-ad28-4a0d-9d2c-f619176037c7\") "
Sep 10 04:59:27.749993 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:27.749971    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf8e8c1-ad28-4a0d-9d2c-f619176037c7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "daf8e8c1-ad28-4a0d-9d2c-f619176037c7" (UID: "daf8e8c1-ad28-4a0d-9d2c-f619176037c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 04:59:27.851117 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:27.851006    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daf8e8c1-ad28-4a0d-9d2c-f619176037c7-maas-api-service-ca\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:59:28.295380 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:28.295341    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:28.555461 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:28.555373    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea9ba2-r6n2l" event={"ID":"daf8e8c1-ad28-4a0d-9d2c-f619176037c7","Type":"ContainerDied","Data":"6242931a478f4be21cbe5dc5aeae99f41db6bcb66ee8f194e49d06e3076cd0f8"}
Sep 10 04:59:28.555461 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:28.555408    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea9ba2-r6n2l"
Sep 10 04:59:28.555461 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:28.555426    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6242931a478f4be21cbe5dc5aeae99f41db6bcb66ee8f194e49d06e3076cd0f8"
Sep 10 04:59:32.126335 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:32.126291    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:34.633646 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:34.633601    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:34.831762 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:34.831728    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:35.332863 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:35.332827    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:35.408036 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:35.408005    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:35.493261 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:35.493221    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:35.563054 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:35.563018    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea9ba2-r6n2l"]
Sep 10 04:59:35.565533 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:35.565499    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:35.566158 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:35.566137    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea9ba2-r6n2l"]
Sep 10 04:59:35.635129 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:35.635009    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:35.862688 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:35.862655    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daf8e8c1-ad28-4a0d-9d2c-f619176037c7" path="/var/lib/kubelet/pods/daf8e8c1-ad28-4a0d-9d2c-f619176037c7/volumes"
Sep 10 04:59:35.963705 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:35.963668    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:36.063346 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:36.063309    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:39.252575 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.252515    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg" podUID="1bcdcb4e-8de9-4158-b164-30df2b15924a" containerName="main" containerID="cri-o://870ddc054cdb87883e5bf9c24f8b0f4c3d2adc368228342154b50f0e5ce73e6a" gracePeriod=45
Sep 10 04:59:39.511258 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.511190    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:39.614737 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.614700    2575 generic.go:356] "Generic (PLEG): container finished" podID="1bcdcb4e-8de9-4158-b164-30df2b15924a" containerID="870ddc054cdb87883e5bf9c24f8b0f4c3d2adc368228342154b50f0e5ce73e6a" exitCode=0
Sep 10 04:59:39.614929 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.614749    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"
Sep 10 04:59:39.614929 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.614784    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg" event={"ID":"1bcdcb4e-8de9-4158-b164-30df2b15924a","Type":"ContainerDied","Data":"870ddc054cdb87883e5bf9c24f8b0f4c3d2adc368228342154b50f0e5ce73e6a"}
Sep 10 04:59:39.614929 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.614829    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg" event={"ID":"1bcdcb4e-8de9-4158-b164-30df2b15924a","Type":"ContainerDied","Data":"e1b07c9ca91eb8cea6b2404083dbf3a02e5709c7953f985b86094a76b8300bfd"}
Sep 10 04:59:39.614929 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.614851    2575 scope.go:117] "RemoveContainer" containerID="870ddc054cdb87883e5bf9c24f8b0f4c3d2adc368228342154b50f0e5ce73e6a"
Sep 10 04:59:39.623620 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.623604    2575 scope.go:117] "RemoveContainer" containerID="870ddc054cdb87883e5bf9c24f8b0f4c3d2adc368228342154b50f0e5ce73e6a"
Sep 10 04:59:39.623871 ip-10-0-132-70 kubenswrapper[2575]: E0910 04:59:39.623854    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"870ddc054cdb87883e5bf9c24f8b0f4c3d2adc368228342154b50f0e5ce73e6a\": container with ID starting with 870ddc054cdb87883e5bf9c24f8b0f4c3d2adc368228342154b50f0e5ce73e6a not found: ID does not exist" containerID="870ddc054cdb87883e5bf9c24f8b0f4c3d2adc368228342154b50f0e5ce73e6a"
Sep 10 04:59:39.623923 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.623880    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"870ddc054cdb87883e5bf9c24f8b0f4c3d2adc368228342154b50f0e5ce73e6a"} err="failed to get container status \"870ddc054cdb87883e5bf9c24f8b0f4c3d2adc368228342154b50f0e5ce73e6a\": rpc error: code = NotFound desc = could not find container \"870ddc054cdb87883e5bf9c24f8b0f4c3d2adc368228342154b50f0e5ce73e6a\": container with ID starting with 870ddc054cdb87883e5bf9c24f8b0f4c3d2adc368228342154b50f0e5ce73e6a not found: ID does not exist"
Sep 10 04:59:39.659740 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.659704    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-tmp-dir\") pod \"1bcdcb4e-8de9-4158-b164-30df2b15924a\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") "
Sep 10 04:59:39.659902 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.659763    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-dshm\") pod \"1bcdcb4e-8de9-4158-b164-30df2b15924a\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") "
Sep 10 04:59:39.659902 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.659784    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-home\") pod \"1bcdcb4e-8de9-4158-b164-30df2b15924a\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") "
Sep 10 04:59:39.659902 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.659807    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-model-cache\") pod \"1bcdcb4e-8de9-4158-b164-30df2b15924a\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") "
Sep 10 04:59:39.659902 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.659853    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjq2z\" (UniqueName: \"kubernetes.io/projected/1bcdcb4e-8de9-4158-b164-30df2b15924a-kube-api-access-zjq2z\") pod \"1bcdcb4e-8de9-4158-b164-30df2b15924a\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") "
Sep 10 04:59:39.660155 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.659954    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bcdcb4e-8de9-4158-b164-30df2b15924a-tls-certs\") pod \"1bcdcb4e-8de9-4158-b164-30df2b15924a\" (UID: \"1bcdcb4e-8de9-4158-b164-30df2b15924a\") "
Sep 10 04:59:39.660155 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.660026    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1bcdcb4e-8de9-4158-b164-30df2b15924a" (UID: "1bcdcb4e-8de9-4158-b164-30df2b15924a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 04:59:39.660155 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.660049    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-home" (OuterVolumeSpecName: "home") pod "1bcdcb4e-8de9-4158-b164-30df2b15924a" (UID: "1bcdcb4e-8de9-4158-b164-30df2b15924a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 04:59:39.660155 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.660089    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-model-cache" (OuterVolumeSpecName: "model-cache") pod "1bcdcb4e-8de9-4158-b164-30df2b15924a" (UID: "1bcdcb4e-8de9-4158-b164-30df2b15924a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 04:59:39.660343 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.660319    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-tmp-dir\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:59:39.660343 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.660339    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-home\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:59:39.660423 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.660352    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-model-cache\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:59:39.662100 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.662057    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bcdcb4e-8de9-4158-b164-30df2b15924a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1bcdcb4e-8de9-4158-b164-30df2b15924a" (UID: "1bcdcb4e-8de9-4158-b164-30df2b15924a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 04:59:39.662269 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.662127    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-dshm" (OuterVolumeSpecName: "dshm") pod "1bcdcb4e-8de9-4158-b164-30df2b15924a" (UID: "1bcdcb4e-8de9-4158-b164-30df2b15924a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 04:59:39.662269 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.662147    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bcdcb4e-8de9-4158-b164-30df2b15924a-kube-api-access-zjq2z" (OuterVolumeSpecName: "kube-api-access-zjq2z") pod "1bcdcb4e-8de9-4158-b164-30df2b15924a" (UID: "1bcdcb4e-8de9-4158-b164-30df2b15924a"). InnerVolumeSpecName "kube-api-access-zjq2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 04:59:39.761178 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.761144    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bcdcb4e-8de9-4158-b164-30df2b15924a-dshm\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:59:39.761178 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.761171    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjq2z\" (UniqueName: \"kubernetes.io/projected/1bcdcb4e-8de9-4158-b164-30df2b15924a-kube-api-access-zjq2z\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:59:39.761178 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.761182    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bcdcb4e-8de9-4158-b164-30df2b15924a-tls-certs\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:59:39.930000 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.929967    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"]
Sep 10 04:59:39.931585 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:39.931563    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-ea9ba2/explicit-model-ea9ba2-kserve-5887b7f897-b49xg"]
Sep 10 04:59:41.862918 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:41.862884    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bcdcb4e-8de9-4158-b164-30df2b15924a" path="/var/lib/kubelet/pods/1bcdcb4e-8de9-4158-b164-30df2b15924a/volumes"
Sep 10 04:59:42.360722 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:42.360682    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:42.428502 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:42.428468    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:42.494972 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:42.494933    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:42.732655 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:42.732615    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aa81ae79-rfjv4"]
Sep 10 04:59:42.733055 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:42.733038    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daf8e8c1-ad28-4a0d-9d2c-f619176037c7" containerName="revoke-keys"
Sep 10 04:59:42.733156 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:42.733057    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="daf8e8c1-ad28-4a0d-9d2c-f619176037c7" containerName="revoke-keys"
Sep 10 04:59:42.733156 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:42.733087    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bcdcb4e-8de9-4158-b164-30df2b15924a" containerName="main"
Sep 10 04:59:42.733156 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:42.733093    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bcdcb4e-8de9-4158-b164-30df2b15924a" containerName="main"
Sep 10 04:59:42.733315 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:42.733173    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bcdcb4e-8de9-4158-b164-30df2b15924a" containerName="main"
Sep 10 04:59:42.733315 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:42.733181    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="daf8e8c1-ad28-4a0d-9d2c-f619176037c7" containerName="revoke-keys"
Sep 10 04:59:42.736391 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:42.736368    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aa81ae79-rfjv4"
Sep 10 04:59:42.738849 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:42.738829    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 04:59:42.739016 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:42.738997    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dkf6w\""
Sep 10 04:59:42.744107 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:42.744084    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aa81ae79-rfjv4"]
Sep 10 04:59:42.754581 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:42.754550    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:42.841758 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:42.841719    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:42.890567 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:42.890532    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a01dd9d-613f-4633-a15f-c10a7839efd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-aa81ae79-rfjv4\" (UID: \"2a01dd9d-613f-4633-a15f-c10a7839efd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aa81ae79-rfjv4"
Sep 10 04:59:42.991832 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:42.991753    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a01dd9d-613f-4633-a15f-c10a7839efd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-aa81ae79-rfjv4\" (UID: \"2a01dd9d-613f-4633-a15f-c10a7839efd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aa81ae79-rfjv4"
Sep 10 04:59:42.992393 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:42.992374    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a01dd9d-613f-4633-a15f-c10a7839efd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-aa81ae79-rfjv4\" (UID: \"2a01dd9d-613f-4633-a15f-c10a7839efd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aa81ae79-rfjv4"
Sep 10 04:59:43.048725 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:43.048697    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aa81ae79-rfjv4"
Sep 10 04:59:43.202760 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:43.202726    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:43.318112 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:43.318056    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aa81ae79-rfjv4"]
Sep 10 04:59:43.320790 ip-10-0-132-70 kubenswrapper[2575]: W0910 04:59:43.320764    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a01dd9d_613f_4633_a15f_c10a7839efd9.slice/crio-54097cab5c235c5bf5f44b9d6404b43dd3958e9b8e2567d72a91cee51de1403d WatchSource:0}: Error finding container 54097cab5c235c5bf5f44b9d6404b43dd3958e9b8e2567d72a91cee51de1403d: Status 404 returned error can't find the container with id 54097cab5c235c5bf5f44b9d6404b43dd3958e9b8e2567d72a91cee51de1403d
Sep 10 04:59:43.516899 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:43.516862    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:43.635601 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:43.635573    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aa81ae79-rfjv4" event={"ID":"2a01dd9d-613f-4633-a15f-c10a7839efd9","Type":"ContainerStarted","Data":"54097cab5c235c5bf5f44b9d6404b43dd3958e9b8e2567d72a91cee51de1403d"}
Sep 10 04:59:43.878079 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:43.878043    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:44.642011 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:44.641976    2575 generic.go:356] "Generic (PLEG): container finished" podID="2a01dd9d-613f-4633-a15f-c10a7839efd9" containerID="2139e10a6d70d18e855d4a00b4fd33f364623e65f52f58aabef6a93201cec917" exitCode=0
Sep 10 04:59:44.642489 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:44.642016    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aa81ae79-rfjv4" event={"ID":"2a01dd9d-613f-4633-a15f-c10a7839efd9","Type":"ContainerDied","Data":"2139e10a6d70d18e855d4a00b4fd33f364623e65f52f58aabef6a93201cec917"}
Sep 10 04:59:45.436143 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:45.436088    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:45.538208 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:45.538153    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:45.639843 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:45.639804    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:45.781241 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:45.781214    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aa81ae79-rfjv4"
Sep 10 04:59:45.921133 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:45.921096    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a01dd9d-613f-4633-a15f-c10a7839efd9-maas-api-service-ca\") pod \"2a01dd9d-613f-4633-a15f-c10a7839efd9\" (UID: \"2a01dd9d-613f-4633-a15f-c10a7839efd9\") "
Sep 10 04:59:45.921437 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:45.921414    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a01dd9d-613f-4633-a15f-c10a7839efd9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2a01dd9d-613f-4633-a15f-c10a7839efd9" (UID: "2a01dd9d-613f-4633-a15f-c10a7839efd9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 04:59:46.022081 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:46.022032    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a01dd9d-613f-4633-a15f-c10a7839efd9-maas-api-service-ca\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 04:59:46.652652 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:46.652569    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aa81ae79-rfjv4"
Sep 10 04:59:46.652798 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:46.652571    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aa81ae79-rfjv4" event={"ID":"2a01dd9d-613f-4633-a15f-c10a7839efd9","Type":"ContainerDied","Data":"54097cab5c235c5bf5f44b9d6404b43dd3958e9b8e2567d72a91cee51de1403d"}
Sep 10 04:59:46.652798 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:46.652692    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54097cab5c235c5bf5f44b9d6404b43dd3958e9b8e2567d72a91cee51de1403d"
Sep 10 04:59:53.180940 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:53.180900    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:53.248150 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:53.248108    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:53.304846 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:53.304810    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:53.359663 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:53.359629    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aa81ae79-rfjv4"]
Sep 10 04:59:53.362021 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:53.361999    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aa81ae79-rfjv4"]
Sep 10 04:59:53.374557 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:53.374527    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:53.739374 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:53.739342    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:53.863391 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:53.863356    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a01dd9d-613f-4633-a15f-c10a7839efd9" path="/var/lib/kubelet/pods/2a01dd9d-613f-4633-a15f-c10a7839efd9/volumes"
Sep 10 04:59:59.502898 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:59.502863    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:59.563215 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:59.563177    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 04:59:59.772437 ip-10-0-132-70 kubenswrapper[2575]: I0910 04:59:59.772351    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:00.108232 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.108146    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:00.139634 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.139601    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a840-29816940-j49vg"]
Sep 10 05:00:00.140023 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.140011    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a01dd9d-613f-4633-a15f-c10a7839efd9" containerName="revoke-keys"
Sep 10 05:00:00.140023 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.140025    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a01dd9d-613f-4633-a15f-c10a7839efd9" containerName="revoke-keys"
Sep 10 05:00:00.140225 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.140120    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a01dd9d-613f-4633-a15f-c10a7839efd9" containerName="revoke-keys"
Sep 10 05:00:00.143355 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.143337    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816940-dfrgf"]
Sep 10 05:00:00.143467 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.143450    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a840-29816940-j49vg"
Sep 10 05:00:00.145905 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.145885    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dkf6w\""
Sep 10 05:00:00.146409 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.146393    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7f2cd3-29816940-5stqb"]
Sep 10 05:00:00.146506 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.146492    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816940-dfrgf"
Sep 10 05:00:00.149464 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.149448    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7f2cd3-29816940-5stqb"
Sep 10 05:00:00.154982 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.154962    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a840-29816940-j49vg"
Sep 10 05:00:00.155526 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.155503    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a840-29816940-j49vg"]
Sep 10 05:00:00.159706 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.158613    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7f2cd3-29816940-5stqb"]
Sep 10 05:00:00.160648 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.160630    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816940-dfrgf"]
Sep 10 05:00:00.166124 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.166103    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816940-dfrgf"
Sep 10 05:00:00.191147 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.191091    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:00.201359 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.201215    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7f2cd3-29816940-5stqb"
Sep 10 05:00:00.331237 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.331201    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:00.443246 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.443220    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a840-29816940-j49vg"]
Sep 10 05:00:00.444902 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:00:00.444877    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8240c76c_132e_435e_871d_d1d8c4c2cd40.slice/crio-0b39aa58830da19de795007dfee23c23bb5b09b1ff4da5abd12bb722e50448c8 WatchSource:0}: Error finding container 0b39aa58830da19de795007dfee23c23bb5b09b1ff4da5abd12bb722e50448c8: Status 404 returned error can't find the container with id 0b39aa58830da19de795007dfee23c23bb5b09b1ff4da5abd12bb722e50448c8
Sep 10 05:00:00.452094 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.452056    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816940-dfrgf"]
Sep 10 05:00:00.453698 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:00:00.453676    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c00d475_da16_4e81_81af_12089a542455.slice/crio-6b4d1172a1079abe00f380c00ef4a99f52d98d640b726115313da19e6db1f318 WatchSource:0}: Error finding container 6b4d1172a1079abe00f380c00ef4a99f52d98d640b726115313da19e6db1f318: Status 404 returned error can't find the container with id 6b4d1172a1079abe00f380c00ef4a99f52d98d640b726115313da19e6db1f318
Sep 10 05:00:00.489975 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.489954    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7f2cd3-29816940-5stqb"]
Sep 10 05:00:00.491960 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:00:00.491935    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133c6cfd_7861_4cc8_b641_775cb00ec15d.slice/crio-017d3bae30c375c328c79b2fec65b1fe83572f33ece6ab64c1df67a7fbe50bd1 WatchSource:0}: Error finding container 017d3bae30c375c328c79b2fec65b1fe83572f33ece6ab64c1df67a7fbe50bd1: Status 404 returned error can't find the container with id 017d3bae30c375c328c79b2fec65b1fe83572f33ece6ab64c1df67a7fbe50bd1
Sep 10 05:00:00.724511 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.724470    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816940-dfrgf" event={"ID":"5c00d475-da16-4e81-81af-12089a542455","Type":"ContainerStarted","Data":"6b4d1172a1079abe00f380c00ef4a99f52d98d640b726115313da19e6db1f318"}
Sep 10 05:00:00.725789 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.725760    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a840-29816940-j49vg" event={"ID":"8240c76c-132e-435e-871d-d1d8c4c2cd40","Type":"ContainerStarted","Data":"0b39aa58830da19de795007dfee23c23bb5b09b1ff4da5abd12bb722e50448c8"}
Sep 10 05:00:00.726910 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:00.726889    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7f2cd3-29816940-5stqb" event={"ID":"133c6cfd-7861-4cc8-b641-775cb00ec15d","Type":"ContainerStarted","Data":"017d3bae30c375c328c79b2fec65b1fe83572f33ece6ab64c1df67a7fbe50bd1"}
Sep 10 05:00:01.733099 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:01.733049    2575 generic.go:356] "Generic (PLEG): container finished" podID="5c00d475-da16-4e81-81af-12089a542455" containerID="1f92a7638663330cc16e02d85a55092423e46716a66c993416029f0e5dedd964" exitCode=0
Sep 10 05:00:01.733510 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:01.733136    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816940-dfrgf" event={"ID":"5c00d475-da16-4e81-81af-12089a542455","Type":"ContainerDied","Data":"1f92a7638663330cc16e02d85a55092423e46716a66c993416029f0e5dedd964"}
Sep 10 05:00:01.734736 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:01.734714    2575 generic.go:356] "Generic (PLEG): container finished" podID="8240c76c-132e-435e-871d-d1d8c4c2cd40" containerID="05336f31a7587f84c75a1208036ecaed4a590eb854cc77fed782eec1a2f36028" exitCode=0
Sep 10 05:00:01.734830 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:01.734741    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a840-29816940-j49vg" event={"ID":"8240c76c-132e-435e-871d-d1d8c4c2cd40","Type":"ContainerDied","Data":"05336f31a7587f84c75a1208036ecaed4a590eb854cc77fed782eec1a2f36028"}
Sep 10 05:00:01.736235 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:01.736207    2575 generic.go:356] "Generic (PLEG): container finished" podID="133c6cfd-7861-4cc8-b641-775cb00ec15d" containerID="0c5a42098e02d22dbe8e463723154aa635652855b320c89e611b75950b6531e7" exitCode=0
Sep 10 05:00:01.736348 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:01.736239    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7f2cd3-29816940-5stqb" event={"ID":"133c6cfd-7861-4cc8-b641-775cb00ec15d","Type":"ContainerDied","Data":"0c5a42098e02d22dbe8e463723154aa635652855b320c89e611b75950b6531e7"}
Sep 10 05:00:02.884496 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:02.884468    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a840-29816940-j49vg"
Sep 10 05:00:02.932024 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:02.932002    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816940-dfrgf"
Sep 10 05:00:02.935309 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:02.935287    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7f2cd3-29816940-5stqb"
Sep 10 05:00:03.747880 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:03.747847    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7f2cd3-29816940-5stqb"
Sep 10 05:00:03.748084 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:03.747882    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7f2cd3-29816940-5stqb" event={"ID":"133c6cfd-7861-4cc8-b641-775cb00ec15d","Type":"ContainerDied","Data":"017d3bae30c375c328c79b2fec65b1fe83572f33ece6ab64c1df67a7fbe50bd1"}
Sep 10 05:00:03.748084 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:03.747930    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="017d3bae30c375c328c79b2fec65b1fe83572f33ece6ab64c1df67a7fbe50bd1"
Sep 10 05:00:03.749333 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:03.749314    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816940-dfrgf"
Sep 10 05:00:03.749333 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:03.749327    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816940-dfrgf" event={"ID":"5c00d475-da16-4e81-81af-12089a542455","Type":"ContainerDied","Data":"6b4d1172a1079abe00f380c00ef4a99f52d98d640b726115313da19e6db1f318"}
Sep 10 05:00:03.749509 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:03.749357    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b4d1172a1079abe00f380c00ef4a99f52d98d640b726115313da19e6db1f318"
Sep 10 05:00:03.750676 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:03.750656    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a840-29816940-j49vg"
Sep 10 05:00:03.750802 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:03.750648    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a840-29816940-j49vg" event={"ID":"8240c76c-132e-435e-871d-d1d8c4c2cd40","Type":"ContainerDied","Data":"0b39aa58830da19de795007dfee23c23bb5b09b1ff4da5abd12bb722e50448c8"}
Sep 10 05:00:03.750802 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:03.750789    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b39aa58830da19de795007dfee23c23bb5b09b1ff4da5abd12bb722e50448c8"
Sep 10 05:00:06.368946 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:06.368908    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:06.428519 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:06.428488    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:06.617394 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:06.617356    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:09.326595 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:09.326549    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:12.326088 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:12.326033    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:12.447361 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:12.447325    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:12.886188 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:12.886091    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:13.133944 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:13.133893    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:13.482992 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:13.482959    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:14.321412 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:14.321371    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:15.758295 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:15.758248    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:16.193205 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:16.193114    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:16.291439 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:16.291401    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:17.354365 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:17.354326    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:18.276284 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:18.276247    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:18.376040 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:18.376002    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:18.752101 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:18.752052    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:19.406177 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:19.406136    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:40.208920 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:40.208836    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3cc69-58drn"]
Sep 10 05:00:40.215840 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:40.209845    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8240c76c-132e-435e-871d-d1d8c4c2cd40" containerName="cleanup"
Sep 10 05:00:40.215840 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:40.209873    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8240c76c-132e-435e-871d-d1d8c4c2cd40" containerName="cleanup"
Sep 10 05:00:40.215840 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:40.209894    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="133c6cfd-7861-4cc8-b641-775cb00ec15d" containerName="cleanup"
Sep 10 05:00:40.215840 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:40.209905    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="133c6cfd-7861-4cc8-b641-775cb00ec15d" containerName="cleanup"
Sep 10 05:00:40.215840 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:40.209932    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c00d475-da16-4e81-81af-12089a542455" containerName="cleanup"
Sep 10 05:00:40.215840 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:40.209941    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c00d475-da16-4e81-81af-12089a542455" containerName="cleanup"
Sep 10 05:00:40.215840 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:40.210224    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8240c76c-132e-435e-871d-d1d8c4c2cd40" containerName="cleanup"
Sep 10 05:00:40.215840 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:40.210237    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c00d475-da16-4e81-81af-12089a542455" containerName="cleanup"
Sep 10 05:00:40.215840 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:40.210254    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="133c6cfd-7861-4cc8-b641-775cb00ec15d" containerName="cleanup"
Sep 10 05:00:40.216703 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:40.216651    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3cc69-58drn"
Sep 10 05:00:40.219095 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:40.219055    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 05:00:40.219219 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:40.219120    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dkf6w\""
Sep 10 05:00:40.219575 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:40.219550    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3cc69-58drn"]
Sep 10 05:00:40.318031 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:40.317991    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e802a6b2-a4cd-49ca-be47-b1fc9528ecb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f3cc69-58drn\" (UID: \"e802a6b2-a4cd-49ca-be47-b1fc9528ecb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3cc69-58drn"
Sep 10 05:00:40.331260 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:40.331224    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:40.418688 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:40.418650    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e802a6b2-a4cd-49ca-be47-b1fc9528ecb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f3cc69-58drn\" (UID: \"e802a6b2-a4cd-49ca-be47-b1fc9528ecb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3cc69-58drn"
Sep 10 05:00:40.419279 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:40.419259    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e802a6b2-a4cd-49ca-be47-b1fc9528ecb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f3cc69-58drn\" (UID: \"e802a6b2-a4cd-49ca-be47-b1fc9528ecb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3cc69-58drn"
Sep 10 05:00:40.529757 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:40.529672    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3cc69-58drn"
Sep 10 05:00:40.795776 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:40.795746    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3cc69-58drn"]
Sep 10 05:00:40.797689 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:00:40.797654    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode802a6b2_a4cd_49ca_be47_b1fc9528ecb2.slice/crio-6454c93f98be1cfa2f765545fb5d2c4bf798cf9ac8785fc70a7160c93332f9e8 WatchSource:0}: Error finding container 6454c93f98be1cfa2f765545fb5d2c4bf798cf9ac8785fc70a7160c93332f9e8: Status 404 returned error can't find the container with id 6454c93f98be1cfa2f765545fb5d2c4bf798cf9ac8785fc70a7160c93332f9e8
Sep 10 05:00:40.946154 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:40.946107    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3cc69-58drn" event={"ID":"e802a6b2-a4cd-49ca-be47-b1fc9528ecb2","Type":"ContainerStarted","Data":"6454c93f98be1cfa2f765545fb5d2c4bf798cf9ac8785fc70a7160c93332f9e8"}
Sep 10 05:00:41.282110 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:41.282048    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:41.529831 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:41.529792    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:41.829496 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:41.829370    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:41.953199 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:41.953162    2575 generic.go:356] "Generic (PLEG): container finished" podID="e802a6b2-a4cd-49ca-be47-b1fc9528ecb2" containerID="668550ae2bf7f48d60e24422e1ddbbe1178ef6509d8e61fd2fb135c591b2452a" exitCode=0
Sep 10 05:00:41.953384 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:41.953246    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3cc69-58drn" event={"ID":"e802a6b2-a4cd-49ca-be47-b1fc9528ecb2","Type":"ContainerDied","Data":"668550ae2bf7f48d60e24422e1ddbbe1178ef6509d8e61fd2fb135c591b2452a"}
Sep 10 05:00:42.285500 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:42.285465    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:42.423035 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:42.423000    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:42.650174 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:42.650083    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:42.783409 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:42.783376    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:43.088843 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:43.088822    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3cc69-58drn"
Sep 10 05:00:43.143377 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:43.143346    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e802a6b2-a4cd-49ca-be47-b1fc9528ecb2-maas-api-service-ca\") pod \"e802a6b2-a4cd-49ca-be47-b1fc9528ecb2\" (UID: \"e802a6b2-a4cd-49ca-be47-b1fc9528ecb2\") "
Sep 10 05:00:43.143675 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:43.143654    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e802a6b2-a4cd-49ca-be47-b1fc9528ecb2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e802a6b2-a4cd-49ca-be47-b1fc9528ecb2" (UID: "e802a6b2-a4cd-49ca-be47-b1fc9528ecb2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:00:43.244453 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:43.244360    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e802a6b2-a4cd-49ca-be47-b1fc9528ecb2-maas-api-service-ca\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:00:43.965081 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:43.965048    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3cc69-58drn"
Sep 10 05:00:43.965541 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:43.965100    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3cc69-58drn" event={"ID":"e802a6b2-a4cd-49ca-be47-b1fc9528ecb2","Type":"ContainerDied","Data":"6454c93f98be1cfa2f765545fb5d2c4bf798cf9ac8785fc70a7160c93332f9e8"}
Sep 10 05:00:43.965541 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:43.965150    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6454c93f98be1cfa2f765545fb5d2c4bf798cf9ac8785fc70a7160c93332f9e8"
Sep 10 05:00:45.343164 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:45.343125    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:45.438146 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:45.438105    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:46.553449 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:46.553406    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:46.653526 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:46.653449    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:46.809131 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:46.809051    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:47.040910 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.040879    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:47.143358 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.143318    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:47.169327 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.169291    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx"]
Sep 10 05:00:47.169729 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.169710    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e802a6b2-a4cd-49ca-be47-b1fc9528ecb2" containerName="revoke-keys"
Sep 10 05:00:47.169816 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.169732    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e802a6b2-a4cd-49ca-be47-b1fc9528ecb2" containerName="revoke-keys"
Sep 10 05:00:47.169877 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.169826    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e802a6b2-a4cd-49ca-be47-b1fc9528ecb2" containerName="revoke-keys"
Sep 10 05:00:47.173012 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.172985    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx"
Sep 10 05:00:47.175459 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.175423    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-97ffde-dockercfg-dj6x4\""
Sep 10 05:00:47.175753 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.175736    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-97ffde\""
Sep 10 05:00:47.191832 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.191810    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx"]
Sep 10 05:00:47.279529 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.279492    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291d939f-8e6b-49cd-bc39-478c9f952983-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx\" (UID: \"291d939f-8e6b-49cd-bc39-478c9f952983\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx"
Sep 10 05:00:47.279686 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.279554    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d96zs\" (UniqueName: \"kubernetes.io/projected/291d939f-8e6b-49cd-bc39-478c9f952983-kube-api-access-d96zs\") pod \"payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx\" (UID: \"291d939f-8e6b-49cd-bc39-478c9f952983\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx"
Sep 10 05:00:47.381054 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.380947    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291d939f-8e6b-49cd-bc39-478c9f952983-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx\" (UID: \"291d939f-8e6b-49cd-bc39-478c9f952983\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx"
Sep 10 05:00:47.381054 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.381018    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d96zs\" (UniqueName: \"kubernetes.io/projected/291d939f-8e6b-49cd-bc39-478c9f952983-kube-api-access-d96zs\") pod \"payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx\" (UID: \"291d939f-8e6b-49cd-bc39-478c9f952983\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx"
Sep 10 05:00:47.381673 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.381654    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291d939f-8e6b-49cd-bc39-478c9f952983-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx\" (UID: \"291d939f-8e6b-49cd-bc39-478c9f952983\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx"
Sep 10 05:00:47.388618 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.388594    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d96zs\" (UniqueName: \"kubernetes.io/projected/291d939f-8e6b-49cd-bc39-478c9f952983-kube-api-access-d96zs\") pod \"payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx\" (UID: \"291d939f-8e6b-49cd-bc39-478c9f952983\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx"
Sep 10 05:00:47.483400 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.483361    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx"
Sep 10 05:00:47.614634 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.614595    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:47.755705 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.755681    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx"]
Sep 10 05:00:47.757809 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:00:47.757781    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291d939f_8e6b_49cd_bc39_478c9f952983.slice/crio-fd4e707ce3f7f77aed75234eb730f4f1da34a47d35afb962d1b5ff3e658e3e4d WatchSource:0}: Error finding container fd4e707ce3f7f77aed75234eb730f4f1da34a47d35afb962d1b5ff3e658e3e4d: Status 404 returned error can't find the container with id fd4e707ce3f7f77aed75234eb730f4f1da34a47d35afb962d1b5ff3e658e3e4d
Sep 10 05:00:47.967524 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.967488    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:47.988934 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.988906    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx" event={"ID":"291d939f-8e6b-49cd-bc39-478c9f952983","Type":"ContainerStarted","Data":"fe85cf863ac84e920aabeb1483cd051365b5fa61bbe5ce750c4cff87da5973a9"}
Sep 10 05:00:47.989123 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.988952    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx" event={"ID":"291d939f-8e6b-49cd-bc39-478c9f952983","Type":"ContainerStarted","Data":"fd4e707ce3f7f77aed75234eb730f4f1da34a47d35afb962d1b5ff3e658e3e4d"}
Sep 10 05:00:47.989123 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:47.988987    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx"
Sep 10 05:00:48.005232 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:48.005142    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx" podStartSLOduration=1.005125616 podStartE2EDuration="1.005125616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:00:48.00356506 +0000 UTC m=+1184.765334251" watchObservedRunningTime="2026-09-10 05:00:48.005125616 +0000 UTC m=+1184.766894798"
Sep 10 05:00:48.418397 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:48.418304    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:50.340513 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:50.340475    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:50.915855 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:50.915815    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:50.932996 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:50.932959    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp"]
Sep 10 05:00:50.933327 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:50.933294    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp" podUID="13e5151f-e68c-4367-8315-a47724341cfd" containerName="payload-processing" containerID="cri-o://d9fb0d736b6a979f8e9e9ffd748e644a36dafa2a8ffa076f94f693712d2a0f37" gracePeriod=30
Sep 10 05:00:50.942518 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:50.941202    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656"]
Sep 10 05:00:50.942518 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:50.941943    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656" podUID="5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6" containerName="payload-pre-processing" containerID="cri-o://a73a8877db1876ed55fb5f4c82796ebee941d1f8ea1db4aeebaa0cb1e9141556" gracePeriod=30
Sep 10 05:00:50.975143 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:50.975112    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:51.089945 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:51.089883    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:51.091707 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:51.091632    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3cc69-58drn"]
Sep 10 05:00:51.093513 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:51.093489    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3cc69-58drn"]
Sep 10 05:00:51.196117 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:51.196089    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp"
Sep 10 05:00:51.223254 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:51.223233    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656"
Sep 10 05:00:51.316543 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:51.316512    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2ttf\" (UniqueName: \"kubernetes.io/projected/13e5151f-e68c-4367-8315-a47724341cfd-kube-api-access-n2ttf\") pod \"13e5151f-e68c-4367-8315-a47724341cfd\" (UID: \"13e5151f-e68c-4367-8315-a47724341cfd\") "
Sep 10 05:00:51.316690 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:51.316643    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e5151f-e68c-4367-8315-a47724341cfd-plugins-config-volume\") pod \"13e5151f-e68c-4367-8315-a47724341cfd\" (UID: \"13e5151f-e68c-4367-8315-a47724341cfd\") "
Sep 10 05:00:51.316690 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:51.316668    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6-plugins-config-volume\") pod \"5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6\" (UID: \"5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6\") "
Sep 10 05:00:51.316797 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:51.316698    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpr2v\" (UniqueName: \"kubernetes.io/projected/5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6-kube-api-access-wpr2v\") pod \"5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6\" (UID: \"5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6\") "
Sep 10 05:00:51.317087 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:51.317030    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6" (UID: "5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:00:51.317221 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:51.317034    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e5151f-e68c-4367-8315-a47724341cfd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13e5151f-e68c-4367-8315-a47724341cfd" (UID: "13e5151f-e68c-4367-8315-a47724341cfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:00:51.318563 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:51.318538    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e5151f-e68c-4367-8315-a47724341cfd-kube-api-access-n2ttf" (OuterVolumeSpecName: "kube-api-access-n2ttf") pod "13e5151f-e68c-4367-8315-a47724341cfd" (UID: "13e5151f-e68c-4367-8315-a47724341cfd"). InnerVolumeSpecName "kube-api-access-n2ttf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:00:51.318645 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:51.318599    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6-kube-api-access-wpr2v" (OuterVolumeSpecName: "kube-api-access-wpr2v") pod "5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6" (UID: "5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6"). InnerVolumeSpecName "kube-api-access-wpr2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:00:51.345045 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:51.345019    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:51.417845 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:51.417815    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e5151f-e68c-4367-8315-a47724341cfd-plugins-config-volume\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:00:51.417845 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:51.417841    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6-plugins-config-volume\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:00:51.417845 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:51.417851    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpr2v\" (UniqueName: \"kubernetes.io/projected/5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6-kube-api-access-wpr2v\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:00:51.418018 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:51.417860    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2ttf\" (UniqueName: \"kubernetes.io/projected/13e5151f-e68c-4367-8315-a47724341cfd-kube-api-access-n2ttf\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:00:51.452403 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:51.452376    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:51.863249 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:51.863172    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e802a6b2-a4cd-49ca-be47-b1fc9528ecb2" path="/var/lib/kubelet/pods/e802a6b2-a4cd-49ca-be47-b1fc9528ecb2/volumes"
Sep 10 05:00:52.011198 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:52.011163    2575 generic.go:356] "Generic (PLEG): container finished" podID="5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6" containerID="a73a8877db1876ed55fb5f4c82796ebee941d1f8ea1db4aeebaa0cb1e9141556" exitCode=0
Sep 10 05:00:52.011355 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:52.011206    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656"
Sep 10 05:00:52.011355 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:52.011249    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656" event={"ID":"5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6","Type":"ContainerDied","Data":"a73a8877db1876ed55fb5f4c82796ebee941d1f8ea1db4aeebaa0cb1e9141556"}
Sep 10 05:00:52.011355 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:52.011302    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656" event={"ID":"5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6","Type":"ContainerDied","Data":"bb6adefd12fc0fabc3b77028a6a642f830bda9174809b66f0adecd895aa3cbe9"}
Sep 10 05:00:52.011355 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:52.011331    2575 scope.go:117] "RemoveContainer" containerID="a73a8877db1876ed55fb5f4c82796ebee941d1f8ea1db4aeebaa0cb1e9141556"
Sep 10 05:00:52.012793 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:52.012771    2575 generic.go:356] "Generic (PLEG): container finished" podID="13e5151f-e68c-4367-8315-a47724341cfd" containerID="d9fb0d736b6a979f8e9e9ffd748e644a36dafa2a8ffa076f94f693712d2a0f37" exitCode=0
Sep 10 05:00:52.012889 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:52.012800    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp" event={"ID":"13e5151f-e68c-4367-8315-a47724341cfd","Type":"ContainerDied","Data":"d9fb0d736b6a979f8e9e9ffd748e644a36dafa2a8ffa076f94f693712d2a0f37"}
Sep 10 05:00:52.012889 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:52.012813    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp"
Sep 10 05:00:52.012889 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:52.012856    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp" event={"ID":"13e5151f-e68c-4367-8315-a47724341cfd","Type":"ContainerDied","Data":"177b058964a0e52af1284711034a445e9c277baa417cd843ee58b985a05587d3"}
Sep 10 05:00:52.020265 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:52.020249    2575 scope.go:117] "RemoveContainer" containerID="a73a8877db1876ed55fb5f4c82796ebee941d1f8ea1db4aeebaa0cb1e9141556"
Sep 10 05:00:52.020522 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:00:52.020503    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73a8877db1876ed55fb5f4c82796ebee941d1f8ea1db4aeebaa0cb1e9141556\": container with ID starting with a73a8877db1876ed55fb5f4c82796ebee941d1f8ea1db4aeebaa0cb1e9141556 not found: ID does not exist" containerID="a73a8877db1876ed55fb5f4c82796ebee941d1f8ea1db4aeebaa0cb1e9141556"
Sep 10 05:00:52.020599 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:52.020533    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73a8877db1876ed55fb5f4c82796ebee941d1f8ea1db4aeebaa0cb1e9141556"} err="failed to get container status \"a73a8877db1876ed55fb5f4c82796ebee941d1f8ea1db4aeebaa0cb1e9141556\": rpc error: code = NotFound desc = could not find container \"a73a8877db1876ed55fb5f4c82796ebee941d1f8ea1db4aeebaa0cb1e9141556\": container with ID starting with a73a8877db1876ed55fb5f4c82796ebee941d1f8ea1db4aeebaa0cb1e9141556 not found: ID does not exist"
Sep 10 05:00:52.020599 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:52.020556    2575 scope.go:117] "RemoveContainer" containerID="d9fb0d736b6a979f8e9e9ffd748e644a36dafa2a8ffa076f94f693712d2a0f37"
Sep 10 05:00:52.028054 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:52.028038    2575 scope.go:117] "RemoveContainer" containerID="d9fb0d736b6a979f8e9e9ffd748e644a36dafa2a8ffa076f94f693712d2a0f37"
Sep 10 05:00:52.028356 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:00:52.028338    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9fb0d736b6a979f8e9e9ffd748e644a36dafa2a8ffa076f94f693712d2a0f37\": container with ID starting with d9fb0d736b6a979f8e9e9ffd748e644a36dafa2a8ffa076f94f693712d2a0f37 not found: ID does not exist" containerID="d9fb0d736b6a979f8e9e9ffd748e644a36dafa2a8ffa076f94f693712d2a0f37"
Sep 10 05:00:52.028407 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:52.028364    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9fb0d736b6a979f8e9e9ffd748e644a36dafa2a8ffa076f94f693712d2a0f37"} err="failed to get container status \"d9fb0d736b6a979f8e9e9ffd748e644a36dafa2a8ffa076f94f693712d2a0f37\": rpc error: code = NotFound desc = could not find container \"d9fb0d736b6a979f8e9e9ffd748e644a36dafa2a8ffa076f94f693712d2a0f37\": container with ID starting with d9fb0d736b6a979f8e9e9ffd748e644a36dafa2a8ffa076f94f693712d2a0f37 not found: ID does not exist"
Sep 10 05:00:52.028541 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:52.028526    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656"]
Sep 10 05:00:52.032129 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:52.032108    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3cc69-5c6b587cc8-9b656"]
Sep 10 05:00:52.046288 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:52.046265    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp"]
Sep 10 05:00:52.050116 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:52.050097    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3cc69-77cc47584f-xqcbp"]
Sep 10 05:00:53.864009 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:53.863965    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13e5151f-e68c-4367-8315-a47724341cfd" path="/var/lib/kubelet/pods/13e5151f-e68c-4367-8315-a47724341cfd/volumes"
Sep 10 05:00:53.864524 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:53.864499    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6" path="/var/lib/kubelet/pods/5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6/volumes"
Sep 10 05:00:57.146378 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:57.146340    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:57.215723 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:57.215684    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:57.403473 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:57.403364    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:57.743227 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:57.743188    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a840-sw5ww"]
Sep 10 05:00:57.743553 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:57.743541    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13e5151f-e68c-4367-8315-a47724341cfd" containerName="payload-processing"
Sep 10 05:00:57.743616 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:57.743555    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="13e5151f-e68c-4367-8315-a47724341cfd" containerName="payload-processing"
Sep 10 05:00:57.743616 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:57.743573    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6" containerName="payload-pre-processing"
Sep 10 05:00:57.743616 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:57.743579    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6" containerName="payload-pre-processing"
Sep 10 05:00:57.743703 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:57.743638    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a29bfa1-f975-4d5f-be6e-6c4a3db78cf6" containerName="payload-pre-processing"
Sep 10 05:00:57.743703 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:57.743645    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="13e5151f-e68c-4367-8315-a47724341cfd" containerName="payload-processing"
Sep 10 05:00:57.748053 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:57.748036    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a840-sw5ww"
Sep 10 05:00:57.750844 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:57.750817    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 05:00:57.751689 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:57.751667    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dkf6w\""
Sep 10 05:00:57.754574 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:57.754553    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a840-sw5ww"]
Sep 10 05:00:57.847914 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:57.847883    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:00:57.873039 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:57.873010    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c205140-44a6-4533-b72d-9ae18483ac59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c9a840-sw5ww\" (UID: \"1c205140-44a6-4533-b72d-9ae18483ac59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a840-sw5ww"
Sep 10 05:00:57.974260 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:57.974224    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c205140-44a6-4533-b72d-9ae18483ac59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c9a840-sw5ww\" (UID: \"1c205140-44a6-4533-b72d-9ae18483ac59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a840-sw5ww"
Sep 10 05:00:57.974821 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:57.974802    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c205140-44a6-4533-b72d-9ae18483ac59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c9a840-sw5ww\" (UID: \"1c205140-44a6-4533-b72d-9ae18483ac59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a840-sw5ww"
Sep 10 05:00:58.059187 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:58.059103    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a840-sw5ww"
Sep 10 05:00:58.340091 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:58.340048    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a840-sw5ww"]
Sep 10 05:00:58.341849 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:00:58.341817    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c205140_44a6_4533_b72d_9ae18483ac59.slice/crio-2362d64e8e2a66520f6b5264a6b939f660d1771e795facfd41f390e022bd72d4 WatchSource:0}: Error finding container 2362d64e8e2a66520f6b5264a6b939f660d1771e795facfd41f390e022bd72d4: Status 404 returned error can't find the container with id 2362d64e8e2a66520f6b5264a6b939f660d1771e795facfd41f390e022bd72d4
Sep 10 05:00:58.995743 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:58.995711    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx"
Sep 10 05:00:59.053485 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:59.053453    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a840-sw5ww" event={"ID":"1c205140-44a6-4533-b72d-9ae18483ac59","Type":"ContainerStarted","Data":"16e28ade335d04948d5dccfc16e88acec65227b0a42b789d5114f0e140cb3fab"}
Sep 10 05:00:59.053629 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:00:59.053515    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a840-sw5ww" event={"ID":"1c205140-44a6-4533-b72d-9ae18483ac59","Type":"ContainerStarted","Data":"2362d64e8e2a66520f6b5264a6b939f660d1771e795facfd41f390e022bd72d4"}
Sep 10 05:01:00.059801 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:00.059757    2575 generic.go:356] "Generic (PLEG): container finished" podID="1c205140-44a6-4533-b72d-9ae18483ac59" containerID="16e28ade335d04948d5dccfc16e88acec65227b0a42b789d5114f0e140cb3fab" exitCode=0
Sep 10 05:01:00.060206 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:00.059822    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a840-sw5ww" event={"ID":"1c205140-44a6-4533-b72d-9ae18483ac59","Type":"ContainerDied","Data":"16e28ade335d04948d5dccfc16e88acec65227b0a42b789d5114f0e140cb3fab"}
Sep 10 05:01:00.182939 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:00.182914    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a840-sw5ww"
Sep 10 05:01:00.296106 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:00.296053    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c205140-44a6-4533-b72d-9ae18483ac59-maas-api-service-ca\") pod \"1c205140-44a6-4533-b72d-9ae18483ac59\" (UID: \"1c205140-44a6-4533-b72d-9ae18483ac59\") "
Sep 10 05:01:00.296409 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:00.296384    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c205140-44a6-4533-b72d-9ae18483ac59-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1c205140-44a6-4533-b72d-9ae18483ac59" (UID: "1c205140-44a6-4533-b72d-9ae18483ac59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:01:00.397167 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:00.397061    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c205140-44a6-4533-b72d-9ae18483ac59-maas-api-service-ca\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:01.065751 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:01.065719    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a840-sw5ww"
Sep 10 05:01:01.066229 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:01.065750    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a840-sw5ww" event={"ID":"1c205140-44a6-4533-b72d-9ae18483ac59","Type":"ContainerDied","Data":"2362d64e8e2a66520f6b5264a6b939f660d1771e795facfd41f390e022bd72d4"}
Sep 10 05:01:01.066229 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:01.065794    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2362d64e8e2a66520f6b5264a6b939f660d1771e795facfd41f390e022bd72d4"
Sep 10 05:01:02.358572 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:02.358533    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:03.490975 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:03.490944    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:03.743315 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:03.743235    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2hd2_9d909143-4c70-4fe1-900a-3cd520db32e5/ovn-acl-logging/0.log"
Sep 10 05:01:03.855789 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:03.855763    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2hd2_9d909143-4c70-4fe1-900a-3cd520db32e5/ovn-acl-logging/0.log"
Sep 10 05:01:04.702863 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:04.702825    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:04.825302 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:04.825267    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7f2cd3-bwhbb"]
Sep 10 05:01:04.825757 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:04.825738    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c205140-44a6-4533-b72d-9ae18483ac59" containerName="revoke-keys"
Sep 10 05:01:04.825757 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:04.825758    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c205140-44a6-4533-b72d-9ae18483ac59" containerName="revoke-keys"
Sep 10 05:01:04.825888 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:04.825842    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c205140-44a6-4533-b72d-9ae18483ac59" containerName="revoke-keys"
Sep 10 05:01:04.829937 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:04.829912    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7f2cd3-bwhbb"
Sep 10 05:01:04.834340 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:04.834314    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dkf6w\""
Sep 10 05:01:04.834828 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:04.834330    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 05:01:04.836328 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:04.836304    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7f2cd3-bwhbb"]
Sep 10 05:01:04.937117 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:04.937043    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ea429e-b1e4-40b9-9840-35a3dda8e5fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7f2cd3-bwhbb\" (UID: \"47ea429e-b1e4-40b9-9840-35a3dda8e5fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7f2cd3-bwhbb"
Sep 10 05:01:05.036643 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:05.036551    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:05.038361 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:05.038332    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ea429e-b1e4-40b9-9840-35a3dda8e5fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7f2cd3-bwhbb\" (UID: \"47ea429e-b1e4-40b9-9840-35a3dda8e5fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7f2cd3-bwhbb"
Sep 10 05:01:05.038914 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:05.038897    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ea429e-b1e4-40b9-9840-35a3dda8e5fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7f2cd3-bwhbb\" (UID: \"47ea429e-b1e4-40b9-9840-35a3dda8e5fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7f2cd3-bwhbb"
Sep 10 05:01:05.143122 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:05.143088    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7f2cd3-bwhbb"
Sep 10 05:01:05.410947 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:05.410923    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7f2cd3-bwhbb"]
Sep 10 05:01:05.412624 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:01:05.412595    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ea429e_b1e4_40b9_9840_35a3dda8e5fe.slice/crio-f90d727bd9d40b94a7731f52da856997a72734afe30bd641cc1f1942c9659f2d WatchSource:0}: Error finding container f90d727bd9d40b94a7731f52da856997a72734afe30bd641cc1f1942c9659f2d: Status 404 returned error can't find the container with id f90d727bd9d40b94a7731f52da856997a72734afe30bd641cc1f1942c9659f2d
Sep 10 05:01:06.093698 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:06.093659    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7f2cd3-bwhbb" event={"ID":"47ea429e-b1e4-40b9-9840-35a3dda8e5fe","Type":"ContainerStarted","Data":"ef323bdce5e960f1f2249ac68f1422fdd3fc06be9cce4357dbf60ac2ff0437af"}
Sep 10 05:01:06.094147 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:06.093717    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7f2cd3-bwhbb" event={"ID":"47ea429e-b1e4-40b9-9840-35a3dda8e5fe","Type":"ContainerStarted","Data":"f90d727bd9d40b94a7731f52da856997a72734afe30bd641cc1f1942c9659f2d"}
Sep 10 05:01:06.107939 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:06.107889    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7f2cd3-bwhbb" podStartSLOduration=2.107874345 podStartE2EDuration="2.107874345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:01:06.107152118 +0000 UTC m=+1202.868921297" watchObservedRunningTime="2026-09-10 05:01:06.107874345 +0000 UTC m=+1202.869643527"
Sep 10 05:01:07.099981 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:07.099948    2575 generic.go:356] "Generic (PLEG): container finished" podID="47ea429e-b1e4-40b9-9840-35a3dda8e5fe" containerID="ef323bdce5e960f1f2249ac68f1422fdd3fc06be9cce4357dbf60ac2ff0437af" exitCode=0
Sep 10 05:01:07.100451 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:07.100035    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7f2cd3-bwhbb" event={"ID":"47ea429e-b1e4-40b9-9840-35a3dda8e5fe","Type":"ContainerDied","Data":"ef323bdce5e960f1f2249ac68f1422fdd3fc06be9cce4357dbf60ac2ff0437af"}
Sep 10 05:01:07.456027 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:07.455994    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:08.188392 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.188356    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:08.210668 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.210636    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a840-29816940-j49vg"]
Sep 10 05:01:08.216273 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.216244    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a840-29816940-j49vg"]
Sep 10 05:01:08.244105 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.244082    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7f2cd3-bwhbb"
Sep 10 05:01:08.265319 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.265288    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7"]
Sep 10 05:01:08.265543 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.265513    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7" podUID="024dc2ea-998d-459d-bd00-2f2a054ae43e" containerName="payload-processing" containerID="cri-o://4237ceb6dcd8a5393248c5cc04d7a7c7dad298e4c28408d02f3c3361ca9bc687" gracePeriod=30
Sep 10 05:01:08.283792 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.283755    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g"]
Sep 10 05:01:08.284019 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.283992    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g" podUID="b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b" containerName="payload-pre-processing" containerID="cri-o://906085c0a1b45309b424907628234f8ec8182229cc3dcf5d6ff5903a04927965" gracePeriod=30
Sep 10 05:01:08.319585 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.319549    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g" podUID="b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.41:9004: connect: connection refused"
Sep 10 05:01:08.319730 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.319549    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7" podUID="024dc2ea-998d-459d-bd00-2f2a054ae43e" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.42:9004: connect: connection refused"
Sep 10 05:01:08.327687 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.327647    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:08.371775 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.371688    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ea429e-b1e4-40b9-9840-35a3dda8e5fe-maas-api-service-ca\") pod \"47ea429e-b1e4-40b9-9840-35a3dda8e5fe\" (UID: \"47ea429e-b1e4-40b9-9840-35a3dda8e5fe\") "
Sep 10 05:01:08.372004 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.371982    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ea429e-b1e4-40b9-9840-35a3dda8e5fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "47ea429e-b1e4-40b9-9840-35a3dda8e5fe" (UID: "47ea429e-b1e4-40b9-9840-35a3dda8e5fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:01:08.422664 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.422588    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:08.472742 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.472710    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ea429e-b1e4-40b9-9840-35a3dda8e5fe-maas-api-service-ca\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:08.490777 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.490744    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a840-sw5ww"]
Sep 10 05:01:08.495131 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.495104    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a840-sw5ww"]
Sep 10 05:01:08.545292 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.545267    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7"
Sep 10 05:01:08.566988 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.566958    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g"
Sep 10 05:01:08.674666 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.674565    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcgks\" (UniqueName: \"kubernetes.io/projected/b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b-kube-api-access-kcgks\") pod \"b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b\" (UID: \"b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b\") "
Sep 10 05:01:08.674666 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.674611    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b-plugins-config-volume\") pod \"b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b\" (UID: \"b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b\") "
Sep 10 05:01:08.674899 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.674669    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024dc2ea-998d-459d-bd00-2f2a054ae43e-plugins-config-volume\") pod \"024dc2ea-998d-459d-bd00-2f2a054ae43e\" (UID: \"024dc2ea-998d-459d-bd00-2f2a054ae43e\") "
Sep 10 05:01:08.674899 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.674733    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwf8s\" (UniqueName: \"kubernetes.io/projected/024dc2ea-998d-459d-bd00-2f2a054ae43e-kube-api-access-pwf8s\") pod \"024dc2ea-998d-459d-bd00-2f2a054ae43e\" (UID: \"024dc2ea-998d-459d-bd00-2f2a054ae43e\") "
Sep 10 05:01:08.675117 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.675059    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b" (UID: "b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:01:08.675172 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.675092    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024dc2ea-998d-459d-bd00-2f2a054ae43e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "024dc2ea-998d-459d-bd00-2f2a054ae43e" (UID: "024dc2ea-998d-459d-bd00-2f2a054ae43e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:01:08.676774 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.676750    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b-kube-api-access-kcgks" (OuterVolumeSpecName: "kube-api-access-kcgks") pod "b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b" (UID: "b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b"). InnerVolumeSpecName "kube-api-access-kcgks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:01:08.676838 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.676816    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024dc2ea-998d-459d-bd00-2f2a054ae43e-kube-api-access-pwf8s" (OuterVolumeSpecName: "kube-api-access-pwf8s") pod "024dc2ea-998d-459d-bd00-2f2a054ae43e" (UID: "024dc2ea-998d-459d-bd00-2f2a054ae43e"). InnerVolumeSpecName "kube-api-access-pwf8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:01:08.702532 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.702498    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:08.775447 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.775409    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcgks\" (UniqueName: \"kubernetes.io/projected/b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b-kube-api-access-kcgks\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:08.775447 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.775439    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b-plugins-config-volume\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:08.775447 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.775450    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024dc2ea-998d-459d-bd00-2f2a054ae43e-plugins-config-volume\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:08.775686 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.775461    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwf8s\" (UniqueName: \"kubernetes.io/projected/024dc2ea-998d-459d-bd00-2f2a054ae43e-kube-api-access-pwf8s\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:08.844162 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:08.844124    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:09.112253 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.112213    2575 generic.go:356] "Generic (PLEG): container finished" podID="b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b" containerID="906085c0a1b45309b424907628234f8ec8182229cc3dcf5d6ff5903a04927965" exitCode=0
Sep 10 05:01:09.112447 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.112263    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g"
Sep 10 05:01:09.112447 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.112292    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g" event={"ID":"b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b","Type":"ContainerDied","Data":"906085c0a1b45309b424907628234f8ec8182229cc3dcf5d6ff5903a04927965"}
Sep 10 05:01:09.112447 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.112353    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g" event={"ID":"b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b","Type":"ContainerDied","Data":"f8d1fd410cc9c69632a64c5bf10c79948a7e916a1db74948032a265cd054738f"}
Sep 10 05:01:09.112447 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.112375    2575 scope.go:117] "RemoveContainer" containerID="906085c0a1b45309b424907628234f8ec8182229cc3dcf5d6ff5903a04927965"
Sep 10 05:01:09.113853 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.113825    2575 generic.go:356] "Generic (PLEG): container finished" podID="024dc2ea-998d-459d-bd00-2f2a054ae43e" containerID="4237ceb6dcd8a5393248c5cc04d7a7c7dad298e4c28408d02f3c3361ca9bc687" exitCode=0
Sep 10 05:01:09.113964 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.113892    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7"
Sep 10 05:01:09.113964 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.113909    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7" event={"ID":"024dc2ea-998d-459d-bd00-2f2a054ae43e","Type":"ContainerDied","Data":"4237ceb6dcd8a5393248c5cc04d7a7c7dad298e4c28408d02f3c3361ca9bc687"}
Sep 10 05:01:09.113964 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.113961    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7" event={"ID":"024dc2ea-998d-459d-bd00-2f2a054ae43e","Type":"ContainerDied","Data":"8b0cee62cbfbf6e2260a66e779fd589fbd4025626692c009505ca40349579122"}
Sep 10 05:01:09.115313 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.115290    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7f2cd3-bwhbb"
Sep 10 05:01:09.115313 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.115303    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7f2cd3-bwhbb" event={"ID":"47ea429e-b1e4-40b9-9840-35a3dda8e5fe","Type":"ContainerDied","Data":"f90d727bd9d40b94a7731f52da856997a72734afe30bd641cc1f1942c9659f2d"}
Sep 10 05:01:09.115451 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.115346    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f90d727bd9d40b94a7731f52da856997a72734afe30bd641cc1f1942c9659f2d"
Sep 10 05:01:09.124103 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.124080    2575 scope.go:117] "RemoveContainer" containerID="906085c0a1b45309b424907628234f8ec8182229cc3dcf5d6ff5903a04927965"
Sep 10 05:01:09.124479 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:01:09.124457    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"906085c0a1b45309b424907628234f8ec8182229cc3dcf5d6ff5903a04927965\": container with ID starting with 906085c0a1b45309b424907628234f8ec8182229cc3dcf5d6ff5903a04927965 not found: ID does not exist" containerID="906085c0a1b45309b424907628234f8ec8182229cc3dcf5d6ff5903a04927965"
Sep 10 05:01:09.124591 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.124486    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906085c0a1b45309b424907628234f8ec8182229cc3dcf5d6ff5903a04927965"} err="failed to get container status \"906085c0a1b45309b424907628234f8ec8182229cc3dcf5d6ff5903a04927965\": rpc error: code = NotFound desc = could not find container \"906085c0a1b45309b424907628234f8ec8182229cc3dcf5d6ff5903a04927965\": container with ID starting with 906085c0a1b45309b424907628234f8ec8182229cc3dcf5d6ff5903a04927965 not found: ID does not exist"
Sep 10 05:01:09.124591 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.124505    2575 scope.go:117] "RemoveContainer" containerID="4237ceb6dcd8a5393248c5cc04d7a7c7dad298e4c28408d02f3c3361ca9bc687"
Sep 10 05:01:09.133264 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.133243    2575 scope.go:117] "RemoveContainer" containerID="4237ceb6dcd8a5393248c5cc04d7a7c7dad298e4c28408d02f3c3361ca9bc687"
Sep 10 05:01:09.133538 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:01:09.133522    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4237ceb6dcd8a5393248c5cc04d7a7c7dad298e4c28408d02f3c3361ca9bc687\": container with ID starting with 4237ceb6dcd8a5393248c5cc04d7a7c7dad298e4c28408d02f3c3361ca9bc687 not found: ID does not exist" containerID="4237ceb6dcd8a5393248c5cc04d7a7c7dad298e4c28408d02f3c3361ca9bc687"
Sep 10 05:01:09.133586 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.133549    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4237ceb6dcd8a5393248c5cc04d7a7c7dad298e4c28408d02f3c3361ca9bc687"} err="failed to get container status \"4237ceb6dcd8a5393248c5cc04d7a7c7dad298e4c28408d02f3c3361ca9bc687\": rpc error: code = NotFound desc = could not find container \"4237ceb6dcd8a5393248c5cc04d7a7c7dad298e4c28408d02f3c3361ca9bc687\": container with ID starting with 4237ceb6dcd8a5393248c5cc04d7a7c7dad298e4c28408d02f3c3361ca9bc687 not found: ID does not exist"
Sep 10 05:01:09.137401 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.137377    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g"]
Sep 10 05:01:09.143434 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.143411    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9a840-77d984d77d-sr57g"]
Sep 10 05:01:09.158311 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.158283    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7"]
Sep 10 05:01:09.161532 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.161503    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9a840-66b54f7-crhs7"]
Sep 10 05:01:09.692145 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.692110    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"]
Sep 10 05:01:09.787826 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.787784    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:09.863630 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.863594    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="024dc2ea-998d-459d-bd00-2f2a054ae43e" path="/var/lib/kubelet/pods/024dc2ea-998d-459d-bd00-2f2a054ae43e/volumes"
Sep 10 05:01:09.863977 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.863963    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c205140-44a6-4533-b72d-9ae18483ac59" path="/var/lib/kubelet/pods/1c205140-44a6-4533-b72d-9ae18483ac59/volumes"
Sep 10 05:01:09.864256 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.864242    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8240c76c-132e-435e-871d-d1d8c4c2cd40" path="/var/lib/kubelet/pods/8240c76c-132e-435e-871d-d1d8c4c2cd40/volumes"
Sep 10 05:01:09.864573 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:09.864559    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b" path="/var/lib/kubelet/pods/b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b/volumes"
Sep 10 05:01:10.319881 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:10.319852    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:12.867227 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:12.867198    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:12.986095 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:12.983231    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-13af27-j56lh"]
Sep 10 05:01:12.986095 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:12.984061    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b" containerName="payload-pre-processing"
Sep 10 05:01:12.986095 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:12.984108    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b" containerName="payload-pre-processing"
Sep 10 05:01:12.986095 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:12.984119    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="024dc2ea-998d-459d-bd00-2f2a054ae43e" containerName="payload-processing"
Sep 10 05:01:12.986095 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:12.984128    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="024dc2ea-998d-459d-bd00-2f2a054ae43e" containerName="payload-processing"
Sep 10 05:01:12.986095 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:12.984170    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47ea429e-b1e4-40b9-9840-35a3dda8e5fe" containerName="revoke-keys"
Sep 10 05:01:12.986095 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:12.984179    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="47ea429e-b1e4-40b9-9840-35a3dda8e5fe" containerName="revoke-keys"
Sep 10 05:01:12.986095 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:12.984353    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b34a5f6d-c5bf-4859-be36-c77d5d7f6f0b" containerName="payload-pre-processing"
Sep 10 05:01:12.986095 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:12.984365    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="024dc2ea-998d-459d-bd00-2f2a054ae43e" containerName="payload-processing"
Sep 10 05:01:12.986095 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:12.984386    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="47ea429e-b1e4-40b9-9840-35a3dda8e5fe" containerName="revoke-keys"
Sep 10 05:01:12.989955 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:12.989915    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-13af27-j56lh"
Sep 10 05:01:12.992546 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:12.992521    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 05:01:12.992777 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:12.992760    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dkf6w\""
Sep 10 05:01:12.993192 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:12.993171    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-13af27-j56lh"]
Sep 10 05:01:13.114684 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:13.114648    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5806a50-2489-48aa-a06b-b6d54d571cd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-13af27-j56lh\" (UID: \"a5806a50-2489-48aa-a06b-b6d54d571cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-13af27-j56lh"
Sep 10 05:01:13.130326 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:13.130230    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:13.215784 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:13.215749    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5806a50-2489-48aa-a06b-b6d54d571cd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-13af27-j56lh\" (UID: \"a5806a50-2489-48aa-a06b-b6d54d571cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-13af27-j56lh"
Sep 10 05:01:13.216354 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:13.216336    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5806a50-2489-48aa-a06b-b6d54d571cd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-13af27-j56lh\" (UID: \"a5806a50-2489-48aa-a06b-b6d54d571cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-13af27-j56lh"
Sep 10 05:01:13.301507 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:13.301476    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-13af27-j56lh"
Sep 10 05:01:13.570532 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:13.570504    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-13af27-j56lh"]
Sep 10 05:01:13.572713 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:01:13.572682    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5806a50_2489_48aa_a06b_b6d54d571cd2.slice/crio-a81dad943262a5d5a398ee6c2a31d13596e05cfc77749a26d68fccb3779de2d4 WatchSource:0}: Error finding container a81dad943262a5d5a398ee6c2a31d13596e05cfc77749a26d68fccb3779de2d4: Status 404 returned error can't find the container with id a81dad943262a5d5a398ee6c2a31d13596e05cfc77749a26d68fccb3779de2d4
Sep 10 05:01:13.919494 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:13.919459    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:14.145257 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:14.145176    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-13af27-j56lh" event={"ID":"a5806a50-2489-48aa-a06b-b6d54d571cd2","Type":"ContainerStarted","Data":"88acdb212f8584adc358690355fab27678d29c97e8667c3889f511f8dc3f02d3"}
Sep 10 05:01:14.145257 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:14.145234    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-13af27-j56lh" event={"ID":"a5806a50-2489-48aa-a06b-b6d54d571cd2","Type":"ContainerStarted","Data":"a81dad943262a5d5a398ee6c2a31d13596e05cfc77749a26d68fccb3779de2d4"}
Sep 10 05:01:14.159860 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:14.159671    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-13af27-j56lh" podStartSLOduration=2.15965692 podStartE2EDuration="2.15965692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:01:14.159101085 +0000 UTC m=+1210.920870260" watchObservedRunningTime="2026-09-10 05:01:14.15965692 +0000 UTC m=+1210.921426101"
Sep 10 05:01:14.959245 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:14.959208    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:15.151760 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:15.151727    2575 generic.go:356] "Generic (PLEG): container finished" podID="a5806a50-2489-48aa-a06b-b6d54d571cd2" containerID="88acdb212f8584adc358690355fab27678d29c97e8667c3889f511f8dc3f02d3" exitCode=0
Sep 10 05:01:15.151929 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:15.151816    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-13af27-j56lh" event={"ID":"a5806a50-2489-48aa-a06b-b6d54d571cd2","Type":"ContainerDied","Data":"88acdb212f8584adc358690355fab27678d29c97e8667c3889f511f8dc3f02d3"}
Sep 10 05:01:15.519397 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:15.519360    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7f2cd3-29816940-5stqb"]
Sep 10 05:01:15.521839 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:15.521810    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:15.521979 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:15.521960    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7f2cd3-29816940-5stqb"]
Sep 10 05:01:15.548749 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:15.548705    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5"]
Sep 10 05:01:15.548973 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:15.548942    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5" podUID="f6d0590b-d31b-41e8-9e9c-333030a3a96d" containerName="payload-pre-processing" containerID="cri-o://25a07260b665c5157b95d6490090ca397d0dcaaa6ec2cc3c98d9dce5b31c4eed" gracePeriod=30
Sep 10 05:01:15.583350 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:15.583310    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:15.702914 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:15.702882    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7f2cd3-bwhbb"]
Sep 10 05:01:15.707285 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:15.707258    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7f2cd3-bwhbb"]
Sep 10 05:01:15.786838 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:15.786814    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5"
Sep 10 05:01:15.863732 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:15.863701    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="133c6cfd-7861-4cc8-b641-775cb00ec15d" path="/var/lib/kubelet/pods/133c6cfd-7861-4cc8-b641-775cb00ec15d/volumes"
Sep 10 05:01:15.864058 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:15.864043    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47ea429e-b1e4-40b9-9840-35a3dda8e5fe" path="/var/lib/kubelet/pods/47ea429e-b1e4-40b9-9840-35a3dda8e5fe/volumes"
Sep 10 05:01:15.941955 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:15.941924    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d0590b-d31b-41e8-9e9c-333030a3a96d-plugins-config-volume\") pod \"f6d0590b-d31b-41e8-9e9c-333030a3a96d\" (UID: \"f6d0590b-d31b-41e8-9e9c-333030a3a96d\") "
Sep 10 05:01:15.942142 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:15.941996    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kl7n\" (UniqueName: \"kubernetes.io/projected/f6d0590b-d31b-41e8-9e9c-333030a3a96d-kube-api-access-2kl7n\") pod \"f6d0590b-d31b-41e8-9e9c-333030a3a96d\" (UID: \"f6d0590b-d31b-41e8-9e9c-333030a3a96d\") "
Sep 10 05:01:15.942433 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:15.942393    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6d0590b-d31b-41e8-9e9c-333030a3a96d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f6d0590b-d31b-41e8-9e9c-333030a3a96d" (UID: "f6d0590b-d31b-41e8-9e9c-333030a3a96d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:01:15.944184 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:15.944159    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d0590b-d31b-41e8-9e9c-333030a3a96d-kube-api-access-2kl7n" (OuterVolumeSpecName: "kube-api-access-2kl7n") pod "f6d0590b-d31b-41e8-9e9c-333030a3a96d" (UID: "f6d0590b-d31b-41e8-9e9c-333030a3a96d"). InnerVolumeSpecName "kube-api-access-2kl7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:01:16.043609 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:16.043523    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d0590b-d31b-41e8-9e9c-333030a3a96d-plugins-config-volume\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:16.043609 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:16.043554    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kl7n\" (UniqueName: \"kubernetes.io/projected/f6d0590b-d31b-41e8-9e9c-333030a3a96d-kube-api-access-2kl7n\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:16.157975 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:16.157941    2575 generic.go:356] "Generic (PLEG): container finished" podID="f6d0590b-d31b-41e8-9e9c-333030a3a96d" containerID="25a07260b665c5157b95d6490090ca397d0dcaaa6ec2cc3c98d9dce5b31c4eed" exitCode=0
Sep 10 05:01:16.157975 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:16.157976    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5"
Sep 10 05:01:16.158262 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:16.158026    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5" event={"ID":"f6d0590b-d31b-41e8-9e9c-333030a3a96d","Type":"ContainerDied","Data":"25a07260b665c5157b95d6490090ca397d0dcaaa6ec2cc3c98d9dce5b31c4eed"}
Sep 10 05:01:16.158262 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:16.158104    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5" event={"ID":"f6d0590b-d31b-41e8-9e9c-333030a3a96d","Type":"ContainerDied","Data":"ac331ed90d72480e6256de562be96158533ee81ecac17100b488d252572a304d"}
Sep 10 05:01:16.158262 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:16.158138    2575 scope.go:117] "RemoveContainer" containerID="25a07260b665c5157b95d6490090ca397d0dcaaa6ec2cc3c98d9dce5b31c4eed"
Sep 10 05:01:16.167403 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:16.167277    2575 scope.go:117] "RemoveContainer" containerID="25a07260b665c5157b95d6490090ca397d0dcaaa6ec2cc3c98d9dce5b31c4eed"
Sep 10 05:01:16.167685 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:01:16.167662    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25a07260b665c5157b95d6490090ca397d0dcaaa6ec2cc3c98d9dce5b31c4eed\": container with ID starting with 25a07260b665c5157b95d6490090ca397d0dcaaa6ec2cc3c98d9dce5b31c4eed not found: ID does not exist" containerID="25a07260b665c5157b95d6490090ca397d0dcaaa6ec2cc3c98d9dce5b31c4eed"
Sep 10 05:01:16.167776 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:16.167691    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25a07260b665c5157b95d6490090ca397d0dcaaa6ec2cc3c98d9dce5b31c4eed"} err="failed to get container status \"25a07260b665c5157b95d6490090ca397d0dcaaa6ec2cc3c98d9dce5b31c4eed\": rpc error: code = NotFound desc = could not find container \"25a07260b665c5157b95d6490090ca397d0dcaaa6ec2cc3c98d9dce5b31c4eed\": container with ID starting with 25a07260b665c5157b95d6490090ca397d0dcaaa6ec2cc3c98d9dce5b31c4eed not found: ID does not exist"
Sep 10 05:01:16.181260 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:16.181231    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5"]
Sep 10 05:01:16.188717 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:16.186995    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7f2cd3-669dd86494-w9fv5"]
Sep 10 05:01:16.285577 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:16.285550    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-13af27-j56lh"
Sep 10 05:01:16.341170 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:16.341078    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:16.441532 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:16.441500    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:16.446785 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:16.446767    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5806a50-2489-48aa-a06b-b6d54d571cd2-maas-api-service-ca\") pod \"a5806a50-2489-48aa-a06b-b6d54d571cd2\" (UID: \"a5806a50-2489-48aa-a06b-b6d54d571cd2\") "
Sep 10 05:01:16.447128 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:16.447103    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5806a50-2489-48aa-a06b-b6d54d571cd2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a5806a50-2489-48aa-a06b-b6d54d571cd2" (UID: "a5806a50-2489-48aa-a06b-b6d54d571cd2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:01:16.547481 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:16.547447    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5806a50-2489-48aa-a06b-b6d54d571cd2-maas-api-service-ca\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:17.165805 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:17.165771    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-13af27-j56lh"
Sep 10 05:01:17.166203 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:17.165776    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-13af27-j56lh" event={"ID":"a5806a50-2489-48aa-a06b-b6d54d571cd2","Type":"ContainerDied","Data":"a81dad943262a5d5a398ee6c2a31d13596e05cfc77749a26d68fccb3779de2d4"}
Sep 10 05:01:17.166203 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:17.165906    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a81dad943262a5d5a398ee6c2a31d13596e05cfc77749a26d68fccb3779de2d4"
Sep 10 05:01:17.850084 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:17.850029    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"]
Sep 10 05:01:17.863081 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:17.863036    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6d0590b-d31b-41e8-9e9c-333030a3a96d" path="/var/lib/kubelet/pods/f6d0590b-d31b-41e8-9e9c-333030a3a96d/volumes"
Sep 10 05:01:17.944482 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:17.944452    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:19.661746 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:19.661712    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:20.342814 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:20.342773    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:20.443693 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:20.443658    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:20.543439 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:20.543397    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:20.643478 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:20.643376    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:20.774264 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:20.774231    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:21.136901 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:21.136862    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:21.675787 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:21.675748    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:22.143599 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:22.143563    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:22.227635 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:22.227595    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:23.115543 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:23.115510    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:23.709811 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:23.709764    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:23.766296 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:23.766266    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck"]
Sep 10 05:01:23.766505 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:23.766477    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck" podUID="96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd" containerName="payload-processing" containerID="cri-o://d37c9574670889ddb0fca76b7c0e8162715b93633fab68c6754adc5c71419294" gracePeriod=30
Sep 10 05:01:23.779180 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:23.779150    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c"]
Sep 10 05:01:23.779813 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:23.779540    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c" podUID="88265117-d02a-4700-9d36-d9486f16067e" containerName="payload-pre-processing" containerID="cri-o://ddf2ebd23b7d443b4a5fd04d9950d9bbfd7c2415c9f529dd5f1218ed68ee9eac" gracePeriod=30
Sep 10 05:01:23.898463 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:23.898435    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:23.974737 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:23.974664    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-13af27-j56lh"]
Sep 10 05:01:23.977535 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:23.977508    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-13af27-j56lh"]
Sep 10 05:01:23.978540 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:23.978515    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:24.036551 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.036522    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:24.047308 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.047287    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck"
Sep 10 05:01:24.050023 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.050005    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c"
Sep 10 05:01:24.113000 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.112966    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn6hw\" (UniqueName: \"kubernetes.io/projected/88265117-d02a-4700-9d36-d9486f16067e-kube-api-access-zn6hw\") pod \"88265117-d02a-4700-9d36-d9486f16067e\" (UID: \"88265117-d02a-4700-9d36-d9486f16067e\") "
Sep 10 05:01:24.113203 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.113032    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlw4w\" (UniqueName: \"kubernetes.io/projected/96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd-kube-api-access-qlw4w\") pod \"96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd\" (UID: \"96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd\") "
Sep 10 05:01:24.113203 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.113131    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88265117-d02a-4700-9d36-d9486f16067e-plugins-config-volume\") pod \"88265117-d02a-4700-9d36-d9486f16067e\" (UID: \"88265117-d02a-4700-9d36-d9486f16067e\") "
Sep 10 05:01:24.113203 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.113165    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd-plugins-config-volume\") pod \"96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd\" (UID: \"96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd\") "
Sep 10 05:01:24.113647 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.113616    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88265117-d02a-4700-9d36-d9486f16067e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "88265117-d02a-4700-9d36-d9486f16067e" (UID: "88265117-d02a-4700-9d36-d9486f16067e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:01:24.113771 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.113667    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd" (UID: "96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:01:24.115388 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.115359    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd-kube-api-access-qlw4w" (OuterVolumeSpecName: "kube-api-access-qlw4w") pod "96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd" (UID: "96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd"). InnerVolumeSpecName "kube-api-access-qlw4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:01:24.115495 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.115412    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88265117-d02a-4700-9d36-d9486f16067e-kube-api-access-zn6hw" (OuterVolumeSpecName: "kube-api-access-zn6hw") pod "88265117-d02a-4700-9d36-d9486f16067e" (UID: "88265117-d02a-4700-9d36-d9486f16067e"). InnerVolumeSpecName "kube-api-access-zn6hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:01:24.201637 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.201602    2575 generic.go:356] "Generic (PLEG): container finished" podID="88265117-d02a-4700-9d36-d9486f16067e" containerID="ddf2ebd23b7d443b4a5fd04d9950d9bbfd7c2415c9f529dd5f1218ed68ee9eac" exitCode=0
Sep 10 05:01:24.201811 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.201648    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c"
Sep 10 05:01:24.201811 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.201687    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c" event={"ID":"88265117-d02a-4700-9d36-d9486f16067e","Type":"ContainerDied","Data":"ddf2ebd23b7d443b4a5fd04d9950d9bbfd7c2415c9f529dd5f1218ed68ee9eac"}
Sep 10 05:01:24.201811 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.201759    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c" event={"ID":"88265117-d02a-4700-9d36-d9486f16067e","Type":"ContainerDied","Data":"6abf6342309ed9c5ea9bfd3620aab6b314021b5cf067d18f9fe5c53e74a33444"}
Sep 10 05:01:24.201811 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.201791    2575 scope.go:117] "RemoveContainer" containerID="ddf2ebd23b7d443b4a5fd04d9950d9bbfd7c2415c9f529dd5f1218ed68ee9eac"
Sep 10 05:01:24.203109 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.203091    2575 generic.go:356] "Generic (PLEG): container finished" podID="96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd" containerID="d37c9574670889ddb0fca76b7c0e8162715b93633fab68c6754adc5c71419294" exitCode=0
Sep 10 05:01:24.203212 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.203130    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck"
Sep 10 05:01:24.203212 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.203135    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck" event={"ID":"96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd","Type":"ContainerDied","Data":"d37c9574670889ddb0fca76b7c0e8162715b93633fab68c6754adc5c71419294"}
Sep 10 05:01:24.203212 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.203170    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck" event={"ID":"96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd","Type":"ContainerDied","Data":"b5b275c5f4706a9d6eba05238c2f66fb6fc0ff3c3af2e971c5c1eeeee6494342"}
Sep 10 05:01:24.211456 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.211432    2575 scope.go:117] "RemoveContainer" containerID="ddf2ebd23b7d443b4a5fd04d9950d9bbfd7c2415c9f529dd5f1218ed68ee9eac"
Sep 10 05:01:24.211725 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:01:24.211706    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf2ebd23b7d443b4a5fd04d9950d9bbfd7c2415c9f529dd5f1218ed68ee9eac\": container with ID starting with ddf2ebd23b7d443b4a5fd04d9950d9bbfd7c2415c9f529dd5f1218ed68ee9eac not found: ID does not exist" containerID="ddf2ebd23b7d443b4a5fd04d9950d9bbfd7c2415c9f529dd5f1218ed68ee9eac"
Sep 10 05:01:24.211785 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.211733    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf2ebd23b7d443b4a5fd04d9950d9bbfd7c2415c9f529dd5f1218ed68ee9eac"} err="failed to get container status \"ddf2ebd23b7d443b4a5fd04d9950d9bbfd7c2415c9f529dd5f1218ed68ee9eac\": rpc error: code = NotFound desc = could not find container \"ddf2ebd23b7d443b4a5fd04d9950d9bbfd7c2415c9f529dd5f1218ed68ee9eac\": container with ID starting with ddf2ebd23b7d443b4a5fd04d9950d9bbfd7c2415c9f529dd5f1218ed68ee9eac not found: ID does not exist"
Sep 10 05:01:24.211785 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.211750    2575 scope.go:117] "RemoveContainer" containerID="d37c9574670889ddb0fca76b7c0e8162715b93633fab68c6754adc5c71419294"
Sep 10 05:01:24.213791 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.213771    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88265117-d02a-4700-9d36-d9486f16067e-plugins-config-volume\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:24.213892 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.213792    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd-plugins-config-volume\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:24.213892 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.213803    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn6hw\" (UniqueName: \"kubernetes.io/projected/88265117-d02a-4700-9d36-d9486f16067e-kube-api-access-zn6hw\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:24.213892 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.213811    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlw4w\" (UniqueName: \"kubernetes.io/projected/96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd-kube-api-access-qlw4w\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:24.220000 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.219980    2575 scope.go:117] "RemoveContainer" containerID="d37c9574670889ddb0fca76b7c0e8162715b93633fab68c6754adc5c71419294"
Sep 10 05:01:24.220263 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:01:24.220242    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37c9574670889ddb0fca76b7c0e8162715b93633fab68c6754adc5c71419294\": container with ID starting with d37c9574670889ddb0fca76b7c0e8162715b93633fab68c6754adc5c71419294 not found: ID does not exist" containerID="d37c9574670889ddb0fca76b7c0e8162715b93633fab68c6754adc5c71419294"
Sep 10 05:01:24.220349 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.220266    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37c9574670889ddb0fca76b7c0e8162715b93633fab68c6754adc5c71419294"} err="failed to get container status \"d37c9574670889ddb0fca76b7c0e8162715b93633fab68c6754adc5c71419294\": rpc error: code = NotFound desc = could not find container \"d37c9574670889ddb0fca76b7c0e8162715b93633fab68c6754adc5c71419294\": container with ID starting with d37c9574670889ddb0fca76b7c0e8162715b93633fab68c6754adc5c71419294 not found: ID does not exist"
Sep 10 05:01:24.223286 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.223265    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c"]
Sep 10 05:01:24.225401 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.225353    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-13af27-74b57d49b6-6c49c"]
Sep 10 05:01:24.235507 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.235486    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck"]
Sep 10 05:01:24.245844 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.242820    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-13af27-8d6d7b69f-9j8ck"]
Sep 10 05:01:24.344356 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.344325    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:24.711656 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.711609    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6" podUID="69190cf0-aa53-48ba-81e3-f8bead942f15" containerName="main" containerID="cri-o://079d38dcfa03d78921c8ec983d00ebd35f7612b3932af6f5e5de9715b729360f" gracePeriod=45
Sep 10 05:01:24.880963 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.880925    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:24.975811 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.975754    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 05:01:24.981198 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:24.981165    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:25.021799 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.021771    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-model-cache\") pod \"69190cf0-aa53-48ba-81e3-f8bead942f15\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") "
Sep 10 05:01:25.021959 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.021815    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-home\") pod \"69190cf0-aa53-48ba-81e3-f8bead942f15\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") "
Sep 10 05:01:25.021959 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.021841    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-tmp-dir\") pod \"69190cf0-aa53-48ba-81e3-f8bead942f15\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") "
Sep 10 05:01:25.021959 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.021884    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-dshm\") pod \"69190cf0-aa53-48ba-81e3-f8bead942f15\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") "
Sep 10 05:01:25.021959 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.021922    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxtbz\" (UniqueName: \"kubernetes.io/projected/69190cf0-aa53-48ba-81e3-f8bead942f15-kube-api-access-mxtbz\") pod \"69190cf0-aa53-48ba-81e3-f8bead942f15\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") "
Sep 10 05:01:25.022204 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.021976    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69190cf0-aa53-48ba-81e3-f8bead942f15-tls-certs\") pod \"69190cf0-aa53-48ba-81e3-f8bead942f15\" (UID: \"69190cf0-aa53-48ba-81e3-f8bead942f15\") "
Sep 10 05:01:25.022204 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.022048    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-model-cache" (OuterVolumeSpecName: "model-cache") pod "69190cf0-aa53-48ba-81e3-f8bead942f15" (UID: "69190cf0-aa53-48ba-81e3-f8bead942f15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:01:25.022315 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.022260    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-model-cache\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:25.022315 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.022263    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69190cf0-aa53-48ba-81e3-f8bead942f15" (UID: "69190cf0-aa53-48ba-81e3-f8bead942f15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:01:25.022502 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.022468    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-home" (OuterVolumeSpecName: "home") pod "69190cf0-aa53-48ba-81e3-f8bead942f15" (UID: "69190cf0-aa53-48ba-81e3-f8bead942f15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:01:25.024343 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.024300    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-dshm" (OuterVolumeSpecName: "dshm") pod "69190cf0-aa53-48ba-81e3-f8bead942f15" (UID: "69190cf0-aa53-48ba-81e3-f8bead942f15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:01:25.024447 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.024419    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69190cf0-aa53-48ba-81e3-f8bead942f15-kube-api-access-mxtbz" (OuterVolumeSpecName: "kube-api-access-mxtbz") pod "69190cf0-aa53-48ba-81e3-f8bead942f15" (UID: "69190cf0-aa53-48ba-81e3-f8bead942f15"). InnerVolumeSpecName "kube-api-access-mxtbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:01:25.024598 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.024577    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69190cf0-aa53-48ba-81e3-f8bead942f15-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69190cf0-aa53-48ba-81e3-f8bead942f15" (UID: "69190cf0-aa53-48ba-81e3-f8bead942f15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 05:01:25.113868 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.113829    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:25.122874 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.122843    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69190cf0-aa53-48ba-81e3-f8bead942f15-tls-certs\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:25.122874 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.122869    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-home\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:25.123036 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.122906    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-tmp-dir\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:25.123036 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.122918    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69190cf0-aa53-48ba-81e3-f8bead942f15-dshm\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:25.123036 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.122930    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxtbz\" (UniqueName: \"kubernetes.io/projected/69190cf0-aa53-48ba-81e3-f8bead942f15-kube-api-access-mxtbz\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:25.210685 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.210652    2575 generic.go:356] "Generic (PLEG): container finished" podID="69190cf0-aa53-48ba-81e3-f8bead942f15" containerID="079d38dcfa03d78921c8ec983d00ebd35f7612b3932af6f5e5de9715b729360f" exitCode=0
Sep 10 05:01:25.210848 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.210691    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"
Sep 10 05:01:25.210848 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.210737    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6" event={"ID":"69190cf0-aa53-48ba-81e3-f8bead942f15","Type":"ContainerDied","Data":"079d38dcfa03d78921c8ec983d00ebd35f7612b3932af6f5e5de9715b729360f"}
Sep 10 05:01:25.210848 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.210773    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6" event={"ID":"69190cf0-aa53-48ba-81e3-f8bead942f15","Type":"ContainerDied","Data":"58373ac40ded89547b889441ae32966ed47d69d9a5b8549b6c34bd21899a641a"}
Sep 10 05:01:25.210848 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.210792    2575 scope.go:117] "RemoveContainer" containerID="079d38dcfa03d78921c8ec983d00ebd35f7612b3932af6f5e5de9715b729360f"
Sep 10 05:01:25.220396 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.220375    2575 scope.go:117] "RemoveContainer" containerID="079d38dcfa03d78921c8ec983d00ebd35f7612b3932af6f5e5de9715b729360f"
Sep 10 05:01:25.220660 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:01:25.220639    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"079d38dcfa03d78921c8ec983d00ebd35f7612b3932af6f5e5de9715b729360f\": container with ID starting with 079d38dcfa03d78921c8ec983d00ebd35f7612b3932af6f5e5de9715b729360f not found: ID does not exist" containerID="079d38dcfa03d78921c8ec983d00ebd35f7612b3932af6f5e5de9715b729360f"
Sep 10 05:01:25.220708 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.220669    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"079d38dcfa03d78921c8ec983d00ebd35f7612b3932af6f5e5de9715b729360f"} err="failed to get container status \"079d38dcfa03d78921c8ec983d00ebd35f7612b3932af6f5e5de9715b729360f\": rpc error: code = NotFound desc = could not find container \"079d38dcfa03d78921c8ec983d00ebd35f7612b3932af6f5e5de9715b729360f\": container with ID starting with 079d38dcfa03d78921c8ec983d00ebd35f7612b3932af6f5e5de9715b729360f not found: ID does not exist"
Sep 10 05:01:25.231852 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.231799    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"]
Sep 10 05:01:25.237310 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.237290    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7f2cd3/simulator-w0-7f2cd3-kserve-59b849c7c6-s8jn6"]
Sep 10 05:01:25.443506 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.443469    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:25.503015 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.502928    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:25.531366 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.531335    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj"]
Sep 10 05:01:25.531688 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.531676    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88265117-d02a-4700-9d36-d9486f16067e" containerName="payload-pre-processing"
Sep 10 05:01:25.531732 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.531690    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="88265117-d02a-4700-9d36-d9486f16067e" containerName="payload-pre-processing"
Sep 10 05:01:25.531732 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.531704    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6d0590b-d31b-41e8-9e9c-333030a3a96d" containerName="payload-pre-processing"
Sep 10 05:01:25.531732 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.531710    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6d0590b-d31b-41e8-9e9c-333030a3a96d" containerName="payload-pre-processing"
Sep 10 05:01:25.531732 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.531725    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5806a50-2489-48aa-a06b-b6d54d571cd2" containerName="revoke-keys"
Sep 10 05:01:25.531732 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.531731    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5806a50-2489-48aa-a06b-b6d54d571cd2" containerName="revoke-keys"
Sep 10 05:01:25.531883 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.531742    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd" containerName="payload-processing"
Sep 10 05:01:25.531883 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.531747    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd" containerName="payload-processing"
Sep 10 05:01:25.531883 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.531758    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69190cf0-aa53-48ba-81e3-f8bead942f15" containerName="main"
Sep 10 05:01:25.531883 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.531763    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="69190cf0-aa53-48ba-81e3-f8bead942f15" containerName="main"
Sep 10 05:01:25.531883 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.531811    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6d0590b-d31b-41e8-9e9c-333030a3a96d" containerName="payload-pre-processing"
Sep 10 05:01:25.531883 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.531818    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5806a50-2489-48aa-a06b-b6d54d571cd2" containerName="revoke-keys"
Sep 10 05:01:25.531883 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.531824    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="69190cf0-aa53-48ba-81e3-f8bead942f15" containerName="main"
Sep 10 05:01:25.531883 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.531830    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="88265117-d02a-4700-9d36-d9486f16067e" containerName="payload-pre-processing"
Sep 10 05:01:25.531883 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.531837    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd" containerName="payload-processing"
Sep 10 05:01:25.536226 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.536201    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj"
Sep 10 05:01:25.538770 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.538747    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8fcf3f8b\""
Sep 10 05:01:25.538770 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.538761    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8fcf3f8b-dockercfg-rbqhp\""
Sep 10 05:01:25.545135 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.545112    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj"]
Sep 10 05:01:25.566881 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.566843    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6"]
Sep 10 05:01:25.570692 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.570662    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6"
Sep 10 05:01:25.585045 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.585018    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6"]
Sep 10 05:01:25.626582 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.626548    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt64t\" (UniqueName: \"kubernetes.io/projected/fe6940b7-0ad6-4a76-a69c-8595622d4e1e-kube-api-access-xt64t\") pod \"payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6\" (UID: \"fe6940b7-0ad6-4a76-a69c-8595622d4e1e\") " pod="openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6"
Sep 10 05:01:25.626758 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.626609    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba6690f-275e-47c4-addd-d164d36aba08-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj\" (UID: \"2ba6690f-275e-47c4-addd-d164d36aba08\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj"
Sep 10 05:01:25.626758 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.626629    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sht4\" (UniqueName: \"kubernetes.io/projected/2ba6690f-275e-47c4-addd-d164d36aba08-kube-api-access-4sht4\") pod \"payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj\" (UID: \"2ba6690f-275e-47c4-addd-d164d36aba08\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj"
Sep 10 05:01:25.626758 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.626696    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe6940b7-0ad6-4a76-a69c-8595622d4e1e-plugins-config-volume\") pod \"payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6\" (UID: \"fe6940b7-0ad6-4a76-a69c-8595622d4e1e\") " pod="openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6"
Sep 10 05:01:25.629896 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.629871    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:25.727161 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.727126    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba6690f-275e-47c4-addd-d164d36aba08-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj\" (UID: \"2ba6690f-275e-47c4-addd-d164d36aba08\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj"
Sep 10 05:01:25.727161 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.727162    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sht4\" (UniqueName: \"kubernetes.io/projected/2ba6690f-275e-47c4-addd-d164d36aba08-kube-api-access-4sht4\") pod \"payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj\" (UID: \"2ba6690f-275e-47c4-addd-d164d36aba08\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj"
Sep 10 05:01:25.727686 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.727204    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe6940b7-0ad6-4a76-a69c-8595622d4e1e-plugins-config-volume\") pod \"payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6\" (UID: \"fe6940b7-0ad6-4a76-a69c-8595622d4e1e\") " pod="openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6"
Sep 10 05:01:25.727686 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.727252    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt64t\" (UniqueName: \"kubernetes.io/projected/fe6940b7-0ad6-4a76-a69c-8595622d4e1e-kube-api-access-xt64t\") pod \"payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6\" (UID: \"fe6940b7-0ad6-4a76-a69c-8595622d4e1e\") " pod="openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6"
Sep 10 05:01:25.727897 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.727874    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba6690f-275e-47c4-addd-d164d36aba08-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj\" (UID: \"2ba6690f-275e-47c4-addd-d164d36aba08\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj"
Sep 10 05:01:25.728050 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.728028    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe6940b7-0ad6-4a76-a69c-8595622d4e1e-plugins-config-volume\") pod \"payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6\" (UID: \"fe6940b7-0ad6-4a76-a69c-8595622d4e1e\") " pod="openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6"
Sep 10 05:01:25.736283 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.736259    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sht4\" (UniqueName: \"kubernetes.io/projected/2ba6690f-275e-47c4-addd-d164d36aba08-kube-api-access-4sht4\") pod \"payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj\" (UID: \"2ba6690f-275e-47c4-addd-d164d36aba08\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj"
Sep 10 05:01:25.736381 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.736266    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt64t\" (UniqueName: \"kubernetes.io/projected/fe6940b7-0ad6-4a76-a69c-8595622d4e1e-kube-api-access-xt64t\") pod \"payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6\" (UID: \"fe6940b7-0ad6-4a76-a69c-8595622d4e1e\") " pod="openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6"
Sep 10 05:01:25.846943 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.846845    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj"
Sep 10 05:01:25.851579 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.851551    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:25.863863 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.863833    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69190cf0-aa53-48ba-81e3-f8bead942f15" path="/var/lib/kubelet/pods/69190cf0-aa53-48ba-81e3-f8bead942f15/volumes"
Sep 10 05:01:25.864293 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.864275    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88265117-d02a-4700-9d36-d9486f16067e" path="/var/lib/kubelet/pods/88265117-d02a-4700-9d36-d9486f16067e/volumes"
Sep 10 05:01:25.864630 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.864617    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd" path="/var/lib/kubelet/pods/96cbdc76-0f4f-4f5b-bf3d-6bb4650860bd/volumes"
Sep 10 05:01:25.864936 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.864923    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5806a50-2489-48aa-a06b-b6d54d571cd2" path="/var/lib/kubelet/pods/a5806a50-2489-48aa-a06b-b6d54d571cd2/volumes"
Sep 10 05:01:25.880830 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.880803    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6"
Sep 10 05:01:25.939185 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:25.939155    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:26.123771 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.123747    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj"]
Sep 10 05:01:26.125522 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:01:26.125497    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba6690f_275e_47c4_addd_d164d36aba08.slice/crio-675614f9c9ec470c63283ad9b737f6e73310cfe66d06b763e80c7cf8ccc1fa69 WatchSource:0}: Error finding container 675614f9c9ec470c63283ad9b737f6e73310cfe66d06b763e80c7cf8ccc1fa69: Status 404 returned error can't find the container with id 675614f9c9ec470c63283ad9b737f6e73310cfe66d06b763e80c7cf8ccc1fa69
Sep 10 05:01:26.173940 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.173897    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6"]
Sep 10 05:01:26.176166 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:01:26.176142    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6940b7_0ad6_4a76_a69c_8595622d4e1e.slice/crio-bcf476acdfaa5ec2e03a7b6602c17d01e52bfa7e8e2c51bac27b59d9c8da3681 WatchSource:0}: Error finding container bcf476acdfaa5ec2e03a7b6602c17d01e52bfa7e8e2c51bac27b59d9c8da3681: Status 404 returned error can't find the container with id bcf476acdfaa5ec2e03a7b6602c17d01e52bfa7e8e2c51bac27b59d9c8da3681
Sep 10 05:01:26.220729 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.220697    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6" event={"ID":"fe6940b7-0ad6-4a76-a69c-8595622d4e1e","Type":"ContainerStarted","Data":"bcf476acdfaa5ec2e03a7b6602c17d01e52bfa7e8e2c51bac27b59d9c8da3681"}
Sep 10 05:01:26.222412 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.222381    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj" event={"ID":"2ba6690f-275e-47c4-addd-d164d36aba08","Type":"ContainerStarted","Data":"edf17ca6d033fb20b65c5f8b6f6137d2d24bd9f6fd4be59501bea8654eec9911"}
Sep 10 05:01:26.222537 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.222437    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj" event={"ID":"2ba6690f-275e-47c4-addd-d164d36aba08","Type":"ContainerStarted","Data":"675614f9c9ec470c63283ad9b737f6e73310cfe66d06b763e80c7cf8ccc1fa69"}
Sep 10 05:01:26.222537 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.222502    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj"
Sep 10 05:01:26.239188 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.239076    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj" podStartSLOduration=1.239044991 podStartE2EDuration="1.239044991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:01:26.238812742 +0000 UTC m=+1223.000581928" watchObservedRunningTime="2026-09-10 05:01:26.239044991 +0000 UTC m=+1223.000814173"
Sep 10 05:01:26.436852 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.436746    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:26.531776 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.531736    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:26.536705 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.536677    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn"]
Sep 10 05:01:26.540022 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.540004    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn"
Sep 10 05:01:26.542645 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.542614    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-4bfef6-dockercfg-prfbq\""
Sep 10 05:01:26.542746 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.542683    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-4bfef6\""
Sep 10 05:01:26.552903 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.552881    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn"]
Sep 10 05:01:26.568650 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.568627    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv"]
Sep 10 05:01:26.572309 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.572292    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv"
Sep 10 05:01:26.582581 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.582555    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv"]
Sep 10 05:01:26.634555 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.634523    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fl6g\" (UniqueName: \"kubernetes.io/projected/491145ab-5023-4559-b7ac-b079dd80dfb7-kube-api-access-7fl6g\") pod \"payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn\" (UID: \"491145ab-5023-4559-b7ac-b079dd80dfb7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn"
Sep 10 05:01:26.634682 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.634596    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/491145ab-5023-4559-b7ac-b079dd80dfb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn\" (UID: \"491145ab-5023-4559-b7ac-b079dd80dfb7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn"
Sep 10 05:01:26.634682 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.634624    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27db7247-bb70-4335-a59e-cd68d95d643b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv\" (UID: \"27db7247-bb70-4335-a59e-cd68d95d643b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv"
Sep 10 05:01:26.634753 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.634727    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9x8j\" (UniqueName: \"kubernetes.io/projected/27db7247-bb70-4335-a59e-cd68d95d643b-kube-api-access-r9x8j\") pod \"payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv\" (UID: \"27db7247-bb70-4335-a59e-cd68d95d643b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv"
Sep 10 05:01:26.735493 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.735456    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/491145ab-5023-4559-b7ac-b079dd80dfb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn\" (UID: \"491145ab-5023-4559-b7ac-b079dd80dfb7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn"
Sep 10 05:01:26.735872 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.735504    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27db7247-bb70-4335-a59e-cd68d95d643b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv\" (UID: \"27db7247-bb70-4335-a59e-cd68d95d643b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv"
Sep 10 05:01:26.735872 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.735571    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9x8j\" (UniqueName: \"kubernetes.io/projected/27db7247-bb70-4335-a59e-cd68d95d643b-kube-api-access-r9x8j\") pod \"payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv\" (UID: \"27db7247-bb70-4335-a59e-cd68d95d643b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv"
Sep 10 05:01:26.735872 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.735630    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fl6g\" (UniqueName: \"kubernetes.io/projected/491145ab-5023-4559-b7ac-b079dd80dfb7-kube-api-access-7fl6g\") pod \"payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn\" (UID: \"491145ab-5023-4559-b7ac-b079dd80dfb7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn"
Sep 10 05:01:26.736270 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.736251    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/491145ab-5023-4559-b7ac-b079dd80dfb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn\" (UID: \"491145ab-5023-4559-b7ac-b079dd80dfb7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn"
Sep 10 05:01:26.736386 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.736366    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27db7247-bb70-4335-a59e-cd68d95d643b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv\" (UID: \"27db7247-bb70-4335-a59e-cd68d95d643b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv"
Sep 10 05:01:26.744253 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.744227    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9x8j\" (UniqueName: \"kubernetes.io/projected/27db7247-bb70-4335-a59e-cd68d95d643b-kube-api-access-r9x8j\") pod \"payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv\" (UID: \"27db7247-bb70-4335-a59e-cd68d95d643b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv"
Sep 10 05:01:26.744368 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.744304    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fl6g\" (UniqueName: \"kubernetes.io/projected/491145ab-5023-4559-b7ac-b079dd80dfb7-kube-api-access-7fl6g\") pod \"payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn\" (UID: \"491145ab-5023-4559-b7ac-b079dd80dfb7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn"
Sep 10 05:01:26.825627 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.825592    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:26.853185 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.853156    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn"
Sep 10 05:01:26.884854 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:26.884824    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv"
Sep 10 05:01:27.138825 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:27.133870    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn"]
Sep 10 05:01:27.141283 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:01:27.141252    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491145ab_5023_4559_b7ac_b079dd80dfb7.slice/crio-b9707f8a38d5a56956a02f81f331b9139376dd5214a50880d8a1340f6c88d006 WatchSource:0}: Error finding container b9707f8a38d5a56956a02f81f331b9139376dd5214a50880d8a1340f6c88d006: Status 404 returned error can't find the container with id b9707f8a38d5a56956a02f81f331b9139376dd5214a50880d8a1340f6c88d006
Sep 10 05:01:27.168039 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:27.168009    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv"]
Sep 10 05:01:27.169281 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:01:27.169256    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27db7247_bb70_4335_a59e_cd68d95d643b.slice/crio-887e78ed5b16028af41c227cdda45156fe9feb7cf0edd9d1c65a00b0da9ebf3f WatchSource:0}: Error finding container 887e78ed5b16028af41c227cdda45156fe9feb7cf0edd9d1c65a00b0da9ebf3f: Status 404 returned error can't find the container with id 887e78ed5b16028af41c227cdda45156fe9feb7cf0edd9d1c65a00b0da9ebf3f
Sep 10 05:01:27.232895 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:27.232847    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn" event={"ID":"491145ab-5023-4559-b7ac-b079dd80dfb7","Type":"ContainerStarted","Data":"94f644c6f2ecb14768f5d5ee52f7c362d5f60270e4ae0e557bf0785f3a360daf"}
Sep 10 05:01:27.233223 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:27.233188    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn"
Sep 10 05:01:27.233394 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:27.233367    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn" event={"ID":"491145ab-5023-4559-b7ac-b079dd80dfb7","Type":"ContainerStarted","Data":"b9707f8a38d5a56956a02f81f331b9139376dd5214a50880d8a1340f6c88d006"}
Sep 10 05:01:27.234885 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:27.234858    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6" event={"ID":"fe6940b7-0ad6-4a76-a69c-8595622d4e1e","Type":"ContainerStarted","Data":"158800370eaa422c8a944b5046a4d7f779915b8a6180f24c188476ca57b05717"}
Sep 10 05:01:27.234990 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:27.234923    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6"
Sep 10 05:01:27.237149 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:27.236939    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv" event={"ID":"27db7247-bb70-4335-a59e-cd68d95d643b","Type":"ContainerStarted","Data":"e17a71a28d7e20c64e4c489090fe36cb201d61377a621a8b41cd778842903b90"}
Sep 10 05:01:27.237149 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:27.237006    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv" event={"ID":"27db7247-bb70-4335-a59e-cd68d95d643b","Type":"ContainerStarted","Data":"887e78ed5b16028af41c227cdda45156fe9feb7cf0edd9d1c65a00b0da9ebf3f"}
Sep 10 05:01:27.252528 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:27.252467    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn" podStartSLOduration=1.252444772 podStartE2EDuration="1.252444772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:01:27.250018516 +0000 UTC m=+1224.011787709" watchObservedRunningTime="2026-09-10 05:01:27.252444772 +0000 UTC m=+1224.014213955"
Sep 10 05:01:27.265933 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:27.265875    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv" podStartSLOduration=1.2658577850000001 podStartE2EDuration="1.265857785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:01:27.265848722 +0000 UTC m=+1224.027617914" watchObservedRunningTime="2026-09-10 05:01:27.265857785 +0000 UTC m=+1224.027626966"
Sep 10 05:01:27.284293 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:27.284233    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6" podStartSLOduration=2.284215768 podStartE2EDuration="2.284215768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:01:27.283272484 +0000 UTC m=+1224.045041669" watchObservedRunningTime="2026-09-10 05:01:27.284215768 +0000 UTC m=+1224.045984949"
Sep 10 05:01:27.638153 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:27.638109    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:28.243637 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:28.243598    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv"
Sep 10 05:01:29.881978 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:29.881943    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:29.991993 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:29.991959    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:30.145598 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:30.145515    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:32.870649 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:32.870574    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg" podUID="78e74217-c203-4f2d-b38b-4ef9bbacd9a4" containerName="main" containerID="cri-o://63fd3561dced1fafb350ea3abf45159a60fa956642118ec2903a98a919be60b1" gracePeriod=45
Sep 10 05:01:33.115515 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.115480    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 05:01:33.197142 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.197119    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltrbk\" (UniqueName: \"kubernetes.io/projected/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-kube-api-access-ltrbk\") pod \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") "
Sep 10 05:01:33.197277 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.197178    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-tmp-dir\") pod \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") "
Sep 10 05:01:33.197277 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.197203    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-dshm\") pod \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") "
Sep 10 05:01:33.197361 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.197279    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-model-cache\") pod \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") "
Sep 10 05:01:33.197361 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.197325    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-tls-certs\") pod \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") "
Sep 10 05:01:33.197463 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.197364    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-home\") pod \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\" (UID: \"78e74217-c203-4f2d-b38b-4ef9bbacd9a4\") "
Sep 10 05:01:33.197520 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.197465    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "78e74217-c203-4f2d-b38b-4ef9bbacd9a4" (UID: "78e74217-c203-4f2d-b38b-4ef9bbacd9a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:01:33.197616 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.197586    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "78e74217-c203-4f2d-b38b-4ef9bbacd9a4" (UID: "78e74217-c203-4f2d-b38b-4ef9bbacd9a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:01:33.197738 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.197705    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-home" (OuterVolumeSpecName: "home") pod "78e74217-c203-4f2d-b38b-4ef9bbacd9a4" (UID: "78e74217-c203-4f2d-b38b-4ef9bbacd9a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:01:33.197858 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.197757    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-tmp-dir\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:33.197858 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.197779    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-model-cache\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:33.199447 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.199416    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-dshm" (OuterVolumeSpecName: "dshm") pod "78e74217-c203-4f2d-b38b-4ef9bbacd9a4" (UID: "78e74217-c203-4f2d-b38b-4ef9bbacd9a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:01:33.199447 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.199432    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "78e74217-c203-4f2d-b38b-4ef9bbacd9a4" (UID: "78e74217-c203-4f2d-b38b-4ef9bbacd9a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 05:01:33.199618 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.199598    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-kube-api-access-ltrbk" (OuterVolumeSpecName: "kube-api-access-ltrbk") pod "78e74217-c203-4f2d-b38b-4ef9bbacd9a4" (UID: "78e74217-c203-4f2d-b38b-4ef9bbacd9a4"). InnerVolumeSpecName "kube-api-access-ltrbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:01:33.272633 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.272602    2575 generic.go:356] "Generic (PLEG): container finished" podID="78e74217-c203-4f2d-b38b-4ef9bbacd9a4" containerID="63fd3561dced1fafb350ea3abf45159a60fa956642118ec2903a98a919be60b1" exitCode=0
Sep 10 05:01:33.272785 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.272651    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"
Sep 10 05:01:33.272785 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.272669    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg" event={"ID":"78e74217-c203-4f2d-b38b-4ef9bbacd9a4","Type":"ContainerDied","Data":"63fd3561dced1fafb350ea3abf45159a60fa956642118ec2903a98a919be60b1"}
Sep 10 05:01:33.272785 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.272707    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg" event={"ID":"78e74217-c203-4f2d-b38b-4ef9bbacd9a4","Type":"ContainerDied","Data":"a65c77aff9cedee3d4c00bcfadfcd33d426eede8c0ffcfee8f7f20890fb38af9"}
Sep 10 05:01:33.272785 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.272730    2575 scope.go:117] "RemoveContainer" containerID="63fd3561dced1fafb350ea3abf45159a60fa956642118ec2903a98a919be60b1"
Sep 10 05:01:33.282077 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.282047    2575 scope.go:117] "RemoveContainer" containerID="63fd3561dced1fafb350ea3abf45159a60fa956642118ec2903a98a919be60b1"
Sep 10 05:01:33.282393 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:01:33.282371    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63fd3561dced1fafb350ea3abf45159a60fa956642118ec2903a98a919be60b1\": container with ID starting with 63fd3561dced1fafb350ea3abf45159a60fa956642118ec2903a98a919be60b1 not found: ID does not exist" containerID="63fd3561dced1fafb350ea3abf45159a60fa956642118ec2903a98a919be60b1"
Sep 10 05:01:33.282495 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.282402    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63fd3561dced1fafb350ea3abf45159a60fa956642118ec2903a98a919be60b1"} err="failed to get container status \"63fd3561dced1fafb350ea3abf45159a60fa956642118ec2903a98a919be60b1\": rpc error: code = NotFound desc = could not find container \"63fd3561dced1fafb350ea3abf45159a60fa956642118ec2903a98a919be60b1\": container with ID starting with 63fd3561dced1fafb350ea3abf45159a60fa956642118ec2903a98a919be60b1 not found: ID does not exist"
Sep 10 05:01:33.292960 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.292934    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"]
Sep 10 05:01:33.297028 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.297007    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-13af27/simulator-w4-13af27-kserve-57cd7c8c65-25xtg"]
Sep 10 05:01:33.298670 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.298651    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-tls-certs\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:33.298730 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.298672    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-home\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:33.298730 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.298681    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltrbk\" (UniqueName: \"kubernetes.io/projected/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-kube-api-access-ltrbk\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:33.298730 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.298690    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78e74217-c203-4f2d-b38b-4ef9bbacd9a4-dshm\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:01:33.866026 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:33.865989    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78e74217-c203-4f2d-b38b-4ef9bbacd9a4" path="/var/lib/kubelet/pods/78e74217-c203-4f2d-b38b-4ef9bbacd9a4/volumes"
Sep 10 05:01:37.239764 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:37.239727    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj"
Sep 10 05:01:38.245980 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:38.245940    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6"
Sep 10 05:01:39.250965 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:39.250935    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn"
Sep 10 05:01:39.251435 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:39.251005    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv"
Sep 10 05:01:45.849310 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:45.849277    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:47.538596 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:47.538556    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:47.638111 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:47.638052    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:48.511370 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:48.511328    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:48.706032 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:48.705996    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:48.806830 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:48.806735    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:49.052190 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:49.052147    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:49.364437 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:49.364368    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:49.540083 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:49.540028    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:49.939839 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:49.939783    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:52.544476 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:52.544441    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:52.871627 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:52.871540    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:52.970180 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:52.970141    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:54.033353 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:54.033321    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:54.132115 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:54.132058    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:54.301044 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:54.300950    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:54.503125 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:54.503090    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:54.590971 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:54.590895    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:54.606810 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:54.606781    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf"]
Sep 10 05:01:54.607314 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:54.607299    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78e74217-c203-4f2d-b38b-4ef9bbacd9a4" containerName="main"
Sep 10 05:01:54.607369 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:54.607317    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="78e74217-c203-4f2d-b38b-4ef9bbacd9a4" containerName="main"
Sep 10 05:01:54.607414 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:54.607403    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="78e74217-c203-4f2d-b38b-4ef9bbacd9a4" containerName="main"
Sep 10 05:01:54.610855 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:54.610837    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf"
Sep 10 05:01:54.617722 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:54.617698    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-3b702d\""
Sep 10 05:01:54.618151 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:54.618129    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-3b702d-dockercfg-kxk8g\""
Sep 10 05:01:54.626971 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:54.626943    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf"]
Sep 10 05:01:54.684637 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:54.684603    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b62e0c61-13cd-45a3-9129-9782cdd02a0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf\" (UID: \"b62e0c61-13cd-45a3-9129-9782cdd02a0b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf"
Sep 10 05:01:54.684826 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:54.684750    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28zgx\" (UniqueName: \"kubernetes.io/projected/b62e0c61-13cd-45a3-9129-9782cdd02a0b-kube-api-access-28zgx\") pod \"payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf\" (UID: \"b62e0c61-13cd-45a3-9129-9782cdd02a0b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf"
Sep 10 05:01:54.785961 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:54.785904    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28zgx\" (UniqueName: \"kubernetes.io/projected/b62e0c61-13cd-45a3-9129-9782cdd02a0b-kube-api-access-28zgx\") pod \"payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf\" (UID: \"b62e0c61-13cd-45a3-9129-9782cdd02a0b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf"
Sep 10 05:01:54.785961 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:54.785970    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b62e0c61-13cd-45a3-9129-9782cdd02a0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf\" (UID: \"b62e0c61-13cd-45a3-9129-9782cdd02a0b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf"
Sep 10 05:01:54.786741 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:54.786714    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b62e0c61-13cd-45a3-9129-9782cdd02a0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf\" (UID: \"b62e0c61-13cd-45a3-9129-9782cdd02a0b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf"
Sep 10 05:01:54.793647 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:54.793625    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28zgx\" (UniqueName: \"kubernetes.io/projected/b62e0c61-13cd-45a3-9129-9782cdd02a0b-kube-api-access-28zgx\") pod \"payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf\" (UID: \"b62e0c61-13cd-45a3-9129-9782cdd02a0b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf"
Sep 10 05:01:54.921880 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:54.921786    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf"
Sep 10 05:01:55.190579 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:55.190507    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf"]
Sep 10 05:01:55.192920 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:01:55.192893    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb62e0c61_13cd_45a3_9129_9782cdd02a0b.slice/crio-1817b6c48fa5d903a1b2f4f5c2ea4e4a14ed28427371580445fa4c698fdf956b WatchSource:0}: Error finding container 1817b6c48fa5d903a1b2f4f5c2ea4e4a14ed28427371580445fa4c698fdf956b: Status 404 returned error can't find the container with id 1817b6c48fa5d903a1b2f4f5c2ea4e4a14ed28427371580445fa4c698fdf956b
Sep 10 05:01:55.405724 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:55.405691    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf" event={"ID":"b62e0c61-13cd-45a3-9129-9782cdd02a0b","Type":"ContainerStarted","Data":"1468c72a835383cb4ba454a2b6e583c26cb18d1dc26c51c74c4c28229e4c8203"}
Sep 10 05:01:55.405921 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:55.405747    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf"
Sep 10 05:01:55.405921 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:55.405765    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf" event={"ID":"b62e0c61-13cd-45a3-9129-9782cdd02a0b","Type":"ContainerStarted","Data":"1817b6c48fa5d903a1b2f4f5c2ea4e4a14ed28427371580445fa4c698fdf956b"}
Sep 10 05:01:55.407724 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:55.407698    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:01:55.420444 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:55.420391    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf" podStartSLOduration=1.420371284 podStartE2EDuration="1.420371284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:01:55.420150382 +0000 UTC m=+1252.181919566" watchObservedRunningTime="2026-09-10 05:01:55.420371284 +0000 UTC m=+1252.182140466"
Sep 10 05:01:55.659042 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:01:55.659003    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:02:02.599412 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:02.599367    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:02:03.932911 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:03.932872    2575 scope.go:117] "RemoveContainer" containerID="7ceefdf542739b0f748d40cfb38f8795944a03eee6ef521384ade78b1a82441b"
Sep 10 05:02:06.413504 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:06.413473    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf"
Sep 10 05:02:14.858344 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:14.858299    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:02:17.864312 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:17.864272    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:02:25.980704 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:25.980658    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"]
Sep 10 05:02:25.986361 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:25.986335    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:25.989796 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:25.989772    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8fcf3f8b\"/\"default-dockercfg-pq48p\""
Sep 10 05:02:25.989934 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:25.989816    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8fcf3f8b\"/\"kube-root-ca.crt\""
Sep 10 05:02:25.990011 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:25.989932    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8fcf3f8b\"/\"openshift-service-ca.crt\""
Sep 10 05:02:25.990366 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:25.990347    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8fcf3f8b\"/\"e2e-lifecycle-model-8fcf3f8b-kserve-self-signed-certs\""
Sep 10 05:02:25.992545 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:25.992520    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"]
Sep 10 05:02:26.010384 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:26.010355    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:02:26.062508 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:26.062473    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/604d2850-95e4-4039-b9a9-37f612c4aac1-tls-certs\") pod \"e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") " pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:26.062676 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:26.062522    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-model-cache\") pod \"e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") " pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:26.062676 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:26.062548    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97zlk\" (UniqueName: \"kubernetes.io/projected/604d2850-95e4-4039-b9a9-37f612c4aac1-kube-api-access-97zlk\") pod \"e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") " pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:26.062676 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:26.062584    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-dshm\") pod \"e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") " pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:26.062801 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:26.062677    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-tmp-dir\") pod \"e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") " pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:26.062801 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:26.062725    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-home\") pod \"e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") " pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:26.163794 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:26.163757    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-model-cache\") pod \"e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") " pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:26.163794 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:26.163793    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97zlk\" (UniqueName: \"kubernetes.io/projected/604d2850-95e4-4039-b9a9-37f612c4aac1-kube-api-access-97zlk\") pod \"e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") " pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:26.164057 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:26.163816    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-dshm\") pod \"e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") " pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:26.164057 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:26.163846    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-tmp-dir\") pod \"e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") " pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:26.164057 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:26.163875    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-home\") pod \"e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") " pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:26.164057 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:26.163969    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/604d2850-95e4-4039-b9a9-37f612c4aac1-tls-certs\") pod \"e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") " pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:26.164325 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:26.164277    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-model-cache\") pod \"e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") " pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:26.164325 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:26.164305    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-tmp-dir\") pod \"e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") " pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:26.164396 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:26.164353    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-home\") pod \"e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") " pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:26.166245 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:26.166225    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-dshm\") pod \"e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") " pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:26.166516 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:26.166499    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/604d2850-95e4-4039-b9a9-37f612c4aac1-tls-certs\") pod \"e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") " pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:26.170794 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:26.170771    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97zlk\" (UniqueName: \"kubernetes.io/projected/604d2850-95e4-4039-b9a9-37f612c4aac1-kube-api-access-97zlk\") pod \"e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") " pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:26.297966 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:26.297858    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:26.569046 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:26.569024    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"]
Sep 10 05:02:26.571267 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:02:26.571239    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod604d2850_95e4_4039_b9a9_37f612c4aac1.slice/crio-eef1eff5b5797eedfc0bdceb3fc54da87efd3538afbe8895cfef5a7fc3c85fd8 WatchSource:0}: Error finding container eef1eff5b5797eedfc0bdceb3fc54da87efd3538afbe8895cfef5a7fc3c85fd8: Status 404 returned error can't find the container with id eef1eff5b5797eedfc0bdceb3fc54da87efd3538afbe8895cfef5a7fc3c85fd8
Sep 10 05:02:27.576263 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:27.576228    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms" event={"ID":"604d2850-95e4-4039-b9a9-37f612c4aac1","Type":"ContainerStarted","Data":"cfa30b9ff8345dbedbb368399f012139e36a2ec2129e6f973d7e0988e3bdd55f"}
Sep 10 05:02:27.576263 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:27.576270    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms" event={"ID":"604d2850-95e4-4039-b9a9-37f612c4aac1","Type":"ContainerStarted","Data":"eef1eff5b5797eedfc0bdceb3fc54da87efd3538afbe8895cfef5a7fc3c85fd8"}
Sep 10 05:02:27.592182 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:27.592132    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms" podStartSLOduration=2.592116024 podStartE2EDuration="2.592116024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:02:27.591194925 +0000 UTC m=+1284.352964108" watchObservedRunningTime="2026-09-10 05:02:27.592116024 +0000 UTC m=+1284.353885207"
Sep 10 05:02:30.794618 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:30.794581    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"]
Sep 10 05:02:30.798642 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:30.798621    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:30.802342 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:30.802318    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4bfef6\"/\"kube-root-ca.crt\""
Sep 10 05:02:30.802463 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:30.802330    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4bfef6\"/\"openshift-service-ca.crt\""
Sep 10 05:02:30.802555 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:30.802528    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4bfef6\"/\"default-dockercfg-zzb95\""
Sep 10 05:02:30.802623 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:30.802343    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4bfef6\"/\"auth-test-model-4bfef6-kserve-self-signed-certs\""
Sep 10 05:02:30.808872 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:30.808846    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"]
Sep 10 05:02:30.817238 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:30.817207    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:02:30.907871 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:30.907836    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-tls-certs\") pod \"auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:30.907871 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:30.907873    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-tmp-dir\") pod \"auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:30.908161 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:30.907911    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2g4q\" (UniqueName: \"kubernetes.io/projected/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-kube-api-access-f2g4q\") pod \"auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:30.908161 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:30.907999    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-home\") pod \"auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:30.908161 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:30.908035    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-dshm\") pod \"auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:30.908161 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:30.908109    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-model-cache\") pod \"auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:31.009567 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:31.009526    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-model-cache\") pod \"auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:31.009741 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:31.009622    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-tls-certs\") pod \"auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:31.009741 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:31.009643    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-tmp-dir\") pod \"auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:31.009741 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:31.009665    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2g4q\" (UniqueName: \"kubernetes.io/projected/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-kube-api-access-f2g4q\") pod \"auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:31.009913 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:31.009743    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-home\") pod \"auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:31.009913 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:31.009785    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-dshm\") pod \"auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:31.010013 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:31.009984    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-model-cache\") pod \"auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:31.010222 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:31.010198    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-tmp-dir\") pod \"auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:31.010317 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:31.010204    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-home\") pod \"auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:31.012026 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:31.012004    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-tls-certs\") pod \"auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:31.012440 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:31.012423    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-dshm\") pod \"auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:31.017349 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:31.017328    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2g4q\" (UniqueName: \"kubernetes.io/projected/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-kube-api-access-f2g4q\") pod \"auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:31.109828 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:31.109736    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:31.378348 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:31.378268    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"]
Sep 10 05:02:31.380715 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:02:31.380683    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfbdc46d_97c5_4a4c_9fef_0577fd6adc63.slice/crio-0ed206883728dc12cd929897f09a58bf2ee40d889a13e35d6bb2c19df4e280b6 WatchSource:0}: Error finding container 0ed206883728dc12cd929897f09a58bf2ee40d889a13e35d6bb2c19df4e280b6: Status 404 returned error can't find the container with id 0ed206883728dc12cd929897f09a58bf2ee40d889a13e35d6bb2c19df4e280b6
Sep 10 05:02:31.598909 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:31.598871    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h" event={"ID":"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63","Type":"ContainerStarted","Data":"086d1ac69b8a1df0433c5c407e2a8f34a0d10ecc9eb13db98f410abc92491add"}
Sep 10 05:02:31.598909 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:31.598916    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h" event={"ID":"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63","Type":"ContainerStarted","Data":"0ed206883728dc12cd929897f09a58bf2ee40d889a13e35d6bb2c19df4e280b6"}
Sep 10 05:02:31.616846 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:31.616797    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h" podStartSLOduration=1.616782707 podStartE2EDuration="1.616782707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:02:31.615243889 +0000 UTC m=+1288.377013074" watchObservedRunningTime="2026-09-10 05:02:31.616782707 +0000 UTC m=+1288.378551939"
Sep 10 05:02:35.525220 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:35.525177    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:02:36.298621 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:36.298586    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:36.298824 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:36.298690    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:36.311631 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:36.311596    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:36.635795 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:36.635719    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:02:41.110042 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:41.109997    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:41.110437 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:41.110054    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:41.123271 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:41.123237    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:41.663207 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:41.663177    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:02:50.503498 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.503467    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"]
Sep 10 05:02:50.506698 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.506682    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:02:50.510528 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.510500    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b702d\"/\"auth-test-model-3b702d-kserve-self-signed-certs\""
Sep 10 05:02:50.510661 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.510529    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b702d\"/\"openshift-service-ca.crt\""
Sep 10 05:02:50.510661 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.510503    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b702d\"/\"default-dockercfg-2ln7j\""
Sep 10 05:02:50.510661 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.510615    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b702d\"/\"kube-root-ca.crt\""
Sep 10 05:02:50.518101 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.518057    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"]
Sep 10 05:02:50.524295 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.524259    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:02:50.598341 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.598310    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-home\") pod \"auth-test-model-3b702d-kserve-55c9f695f5-rnl9v\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:02:50.598501 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.598350    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-model-cache\") pod \"auth-test-model-3b702d-kserve-55c9f695f5-rnl9v\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:02:50.598501 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.598440    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-dshm\") pod \"auth-test-model-3b702d-kserve-55c9f695f5-rnl9v\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:02:50.598501 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.598489    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-tmp-dir\") pod \"auth-test-model-3b702d-kserve-55c9f695f5-rnl9v\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:02:50.598643 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.598512    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-tls-certs\") pod \"auth-test-model-3b702d-kserve-55c9f695f5-rnl9v\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:02:50.598643 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.598607    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xr2g\" (UniqueName: \"kubernetes.io/projected/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-kube-api-access-6xr2g\") pod \"auth-test-model-3b702d-kserve-55c9f695f5-rnl9v\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:02:50.699450 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.699413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-dshm\") pod \"auth-test-model-3b702d-kserve-55c9f695f5-rnl9v\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:02:50.699629 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.699459    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-tmp-dir\") pod \"auth-test-model-3b702d-kserve-55c9f695f5-rnl9v\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:02:50.699629 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.699482    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-tls-certs\") pod \"auth-test-model-3b702d-kserve-55c9f695f5-rnl9v\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:02:50.699629 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.699529    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xr2g\" (UniqueName: \"kubernetes.io/projected/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-kube-api-access-6xr2g\") pod \"auth-test-model-3b702d-kserve-55c9f695f5-rnl9v\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:02:50.699629 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.699576    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-home\") pod \"auth-test-model-3b702d-kserve-55c9f695f5-rnl9v\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:02:50.699629 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.699610    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-model-cache\") pod \"auth-test-model-3b702d-kserve-55c9f695f5-rnl9v\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:02:50.699999 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.699973    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-tmp-dir\") pod \"auth-test-model-3b702d-kserve-55c9f695f5-rnl9v\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:02:50.700095 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.700022    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-model-cache\") pod \"auth-test-model-3b702d-kserve-55c9f695f5-rnl9v\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:02:50.700095 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.700007    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-home\") pod \"auth-test-model-3b702d-kserve-55c9f695f5-rnl9v\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:02:50.701747 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.701728    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-dshm\") pod \"auth-test-model-3b702d-kserve-55c9f695f5-rnl9v\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:02:50.702014 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.701984    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-tls-certs\") pod \"auth-test-model-3b702d-kserve-55c9f695f5-rnl9v\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:02:50.707117 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.707095    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xr2g\" (UniqueName: \"kubernetes.io/projected/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-kube-api-access-6xr2g\") pod \"auth-test-model-3b702d-kserve-55c9f695f5-rnl9v\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:02:50.815888 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:50.815812    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:02:51.087627 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:51.087540    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"]
Sep 10 05:02:51.089887 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:02:51.089861    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5508ef62_7e0c_4d0d_8d16_84b55aa30bf6.slice/crio-0919259f115712e6379171a03d16d4dfd44d65000da4bca99de4166d371b4f62 WatchSource:0}: Error finding container 0919259f115712e6379171a03d16d4dfd44d65000da4bca99de4166d371b4f62: Status 404 returned error can't find the container with id 0919259f115712e6379171a03d16d4dfd44d65000da4bca99de4166d371b4f62
Sep 10 05:02:51.703248 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:51.703210    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v" event={"ID":"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6","Type":"ContainerStarted","Data":"b6f650a1124ee1ad847b2b6b3762cf76d16911a369055f0dd8dc2c3ae0d6d29c"}
Sep 10 05:02:51.703248 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:51.703254    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v" event={"ID":"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6","Type":"ContainerStarted","Data":"0919259f115712e6379171a03d16d4dfd44d65000da4bca99de4166d371b4f62"}
Sep 10 05:02:51.719786 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:51.719737    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v" podStartSLOduration=1.719720497 podStartE2EDuration="1.719720497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:02:51.718331254 +0000 UTC m=+1308.480100436" watchObservedRunningTime="2026-09-10 05:02:51.719720497 +0000 UTC m=+1308.481489692"
Sep 10 05:02:54.465390 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:54.465351    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fcf3f8b-4pm2h"]
Sep 10 05:02:54.468723 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:54.468705    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fcf3f8b-4pm2h"
Sep 10 05:02:54.471150 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:54.471126    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 05:02:54.471263 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:54.471135    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dkf6w\""
Sep 10 05:02:54.475312 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:54.475291    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fcf3f8b-4pm2h"]
Sep 10 05:02:54.635984 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:54.635949    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f494fa-d93c-42a2-bb80-de30c0642102-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8fcf3f8b-4pm2h\" (UID: \"f2f494fa-d93c-42a2-bb80-de30c0642102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fcf3f8b-4pm2h"
Sep 10 05:02:54.736920 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:54.736837    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f494fa-d93c-42a2-bb80-de30c0642102-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8fcf3f8b-4pm2h\" (UID: \"f2f494fa-d93c-42a2-bb80-de30c0642102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fcf3f8b-4pm2h"
Sep 10 05:02:54.737458 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:54.737439    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f494fa-d93c-42a2-bb80-de30c0642102-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8fcf3f8b-4pm2h\" (UID: \"f2f494fa-d93c-42a2-bb80-de30c0642102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fcf3f8b-4pm2h"
Sep 10 05:02:54.760947 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:54.760919    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:02:54.780785 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:54.780754    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fcf3f8b-4pm2h"
Sep 10 05:02:55.056912 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:55.056816    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fcf3f8b-4pm2h"]
Sep 10 05:02:55.063626 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:02:55.063592    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f494fa_d93c_42a2_bb80_de30c0642102.slice/crio-637d8aabe451ea44ac1d3d5c3e90afb6fbc724d3c747125c820c3f64e86c2df7 WatchSource:0}: Error finding container 637d8aabe451ea44ac1d3d5c3e90afb6fbc724d3c747125c820c3f64e86c2df7: Status 404 returned error can't find the container with id 637d8aabe451ea44ac1d3d5c3e90afb6fbc724d3c747125c820c3f64e86c2df7
Sep 10 05:02:55.726147 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:55.726108    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fcf3f8b-4pm2h" event={"ID":"f2f494fa-d93c-42a2-bb80-de30c0642102","Type":"ContainerStarted","Data":"e9dff8fd0dc784a6e7ee9f4b2ab159c369177104431897107da91bb07f3dd67e"}
Sep 10 05:02:55.726528 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:55.726194    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fcf3f8b-4pm2h" event={"ID":"f2f494fa-d93c-42a2-bb80-de30c0642102","Type":"ContainerStarted","Data":"637d8aabe451ea44ac1d3d5c3e90afb6fbc724d3c747125c820c3f64e86c2df7"}
Sep 10 05:02:55.740252 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:55.740196    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fcf3f8b-4pm2h" podStartSLOduration=1.740181031 podStartE2EDuration="1.740181031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:02:55.739213929 +0000 UTC m=+1312.500983111" watchObservedRunningTime="2026-09-10 05:02:55.740181031 +0000 UTC m=+1312.501950212"
Sep 10 05:02:56.733018 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:56.732975    2575 generic.go:356] "Generic (PLEG): container finished" podID="f2f494fa-d93c-42a2-bb80-de30c0642102" containerID="e9dff8fd0dc784a6e7ee9f4b2ab159c369177104431897107da91bb07f3dd67e" exitCode=0
Sep 10 05:02:56.733409 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:56.733057    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fcf3f8b-4pm2h" event={"ID":"f2f494fa-d93c-42a2-bb80-de30c0642102","Type":"ContainerDied","Data":"e9dff8fd0dc784a6e7ee9f4b2ab159c369177104431897107da91bb07f3dd67e"}
Sep 10 05:02:57.875691 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:57.875663    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fcf3f8b-4pm2h"
Sep 10 05:02:57.963462 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:57.963425    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f494fa-d93c-42a2-bb80-de30c0642102-maas-api-service-ca\") pod \"f2f494fa-d93c-42a2-bb80-de30c0642102\" (UID: \"f2f494fa-d93c-42a2-bb80-de30c0642102\") "
Sep 10 05:02:57.963798 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:57.963775    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f494fa-d93c-42a2-bb80-de30c0642102-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f2f494fa-d93c-42a2-bb80-de30c0642102" (UID: "f2f494fa-d93c-42a2-bb80-de30c0642102"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:02:58.064395 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:58.064318    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f494fa-d93c-42a2-bb80-de30c0642102-maas-api-service-ca\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:02:58.746296 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:58.746263    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fcf3f8b-4pm2h"
Sep 10 05:02:58.746296 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:58.746280    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fcf3f8b-4pm2h" event={"ID":"f2f494fa-d93c-42a2-bb80-de30c0642102","Type":"ContainerDied","Data":"637d8aabe451ea44ac1d3d5c3e90afb6fbc724d3c747125c820c3f64e86c2df7"}
Sep 10 05:02:58.746526 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:02:58.746333    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="637d8aabe451ea44ac1d3d5c3e90afb6fbc724d3c747125c820c3f64e86c2df7"
Sep 10 05:03:00.816716 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:00.816677    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:03:00.817228 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:00.816732    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:03:00.828943 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:00.828913    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:03:01.772627 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:01.772595    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:03:03.982305 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:03.982272    2575 scope.go:117] "RemoveContainer" containerID="aa9151d1f3cb6ff8e63445fd247370628a38e968269ca88f1c0de3e9a98516bc"
Sep 10 05:03:03.990850 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:03.990830    2575 scope.go:117] "RemoveContainer" containerID="ea5d794542377add928bd12429bdccc7cb3354ba556333c10e24d446082753d6"
Sep 10 05:03:04.389759 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:04.389643    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:03:09.947567 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:09.947525    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:03:09.964101 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:09.964056    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6"]
Sep 10 05:03:09.964342 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:09.964316    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6" podUID="fe6940b7-0ad6-4a76-a69c-8595622d4e1e" containerName="payload-processing" containerID="cri-o://158800370eaa422c8a944b5046a4d7f779915b8a6180f24c188476ca57b05717" gracePeriod=30
Sep 10 05:03:09.972882 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:09.972853    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj"]
Sep 10 05:03:09.973123 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:09.973099    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj" podUID="2ba6690f-275e-47c4-addd-d164d36aba08" containerName="payload-pre-processing" containerID="cri-o://edf17ca6d033fb20b65c5f8b6f6137d2d24bd9f6fd4be59501bea8654eec9911" gracePeriod=30
Sep 10 05:03:10.047303 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.047267    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:03:10.121135 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.121099    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fcf3f8b-4pm2h"]
Sep 10 05:03:10.124486 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.124400    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:03:10.126031 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.126006    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fcf3f8b-4pm2h"]
Sep 10 05:03:10.187824 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.187787    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:03:10.226994 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.226964    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6"
Sep 10 05:03:10.262503 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.262472    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj"
Sep 10 05:03:10.283400 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.283372    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe6940b7-0ad6-4a76-a69c-8595622d4e1e-plugins-config-volume\") pod \"fe6940b7-0ad6-4a76-a69c-8595622d4e1e\" (UID: \"fe6940b7-0ad6-4a76-a69c-8595622d4e1e\") "
Sep 10 05:03:10.283571 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.283489    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xt64t\" (UniqueName: \"kubernetes.io/projected/fe6940b7-0ad6-4a76-a69c-8595622d4e1e-kube-api-access-xt64t\") pod \"fe6940b7-0ad6-4a76-a69c-8595622d4e1e\" (UID: \"fe6940b7-0ad6-4a76-a69c-8595622d4e1e\") "
Sep 10 05:03:10.283846 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.283814    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe6940b7-0ad6-4a76-a69c-8595622d4e1e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe6940b7-0ad6-4a76-a69c-8595622d4e1e" (UID: "fe6940b7-0ad6-4a76-a69c-8595622d4e1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:03:10.285597 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.285569    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe6940b7-0ad6-4a76-a69c-8595622d4e1e-kube-api-access-xt64t" (OuterVolumeSpecName: "kube-api-access-xt64t") pod "fe6940b7-0ad6-4a76-a69c-8595622d4e1e" (UID: "fe6940b7-0ad6-4a76-a69c-8595622d4e1e"). InnerVolumeSpecName "kube-api-access-xt64t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:03:10.383986 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.383950    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sht4\" (UniqueName: \"kubernetes.io/projected/2ba6690f-275e-47c4-addd-d164d36aba08-kube-api-access-4sht4\") pod \"2ba6690f-275e-47c4-addd-d164d36aba08\" (UID: \"2ba6690f-275e-47c4-addd-d164d36aba08\") "
Sep 10 05:03:10.384213 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.384040    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba6690f-275e-47c4-addd-d164d36aba08-plugins-config-volume\") pod \"2ba6690f-275e-47c4-addd-d164d36aba08\" (UID: \"2ba6690f-275e-47c4-addd-d164d36aba08\") "
Sep 10 05:03:10.384314 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.384298    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt64t\" (UniqueName: \"kubernetes.io/projected/fe6940b7-0ad6-4a76-a69c-8595622d4e1e-kube-api-access-xt64t\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:03:10.384372 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.384314    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe6940b7-0ad6-4a76-a69c-8595622d4e1e-plugins-config-volume\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:03:10.384423 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.384391    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba6690f-275e-47c4-addd-d164d36aba08-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2ba6690f-275e-47c4-addd-d164d36aba08" (UID: "2ba6690f-275e-47c4-addd-d164d36aba08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:03:10.386116 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.386092    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba6690f-275e-47c4-addd-d164d36aba08-kube-api-access-4sht4" (OuterVolumeSpecName: "kube-api-access-4sht4") pod "2ba6690f-275e-47c4-addd-d164d36aba08" (UID: "2ba6690f-275e-47c4-addd-d164d36aba08"). InnerVolumeSpecName "kube-api-access-4sht4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:03:10.485564 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.485467    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sht4\" (UniqueName: \"kubernetes.io/projected/2ba6690f-275e-47c4-addd-d164d36aba08-kube-api-access-4sht4\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:03:10.485564 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.485504    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba6690f-275e-47c4-addd-d164d36aba08-plugins-config-volume\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:03:10.815686 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.815600    2575 generic.go:356] "Generic (PLEG): container finished" podID="2ba6690f-275e-47c4-addd-d164d36aba08" containerID="edf17ca6d033fb20b65c5f8b6f6137d2d24bd9f6fd4be59501bea8654eec9911" exitCode=0
Sep 10 05:03:10.815686 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.815644    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj"
Sep 10 05:03:10.815877 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.815679    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj" event={"ID":"2ba6690f-275e-47c4-addd-d164d36aba08","Type":"ContainerDied","Data":"edf17ca6d033fb20b65c5f8b6f6137d2d24bd9f6fd4be59501bea8654eec9911"}
Sep 10 05:03:10.815877 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.815756    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj" event={"ID":"2ba6690f-275e-47c4-addd-d164d36aba08","Type":"ContainerDied","Data":"675614f9c9ec470c63283ad9b737f6e73310cfe66d06b763e80c7cf8ccc1fa69"}
Sep 10 05:03:10.815877 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.815778    2575 scope.go:117] "RemoveContainer" containerID="edf17ca6d033fb20b65c5f8b6f6137d2d24bd9f6fd4be59501bea8654eec9911"
Sep 10 05:03:10.817148 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.817122    2575 generic.go:356] "Generic (PLEG): container finished" podID="fe6940b7-0ad6-4a76-a69c-8595622d4e1e" containerID="158800370eaa422c8a944b5046a4d7f779915b8a6180f24c188476ca57b05717" exitCode=0
Sep 10 05:03:10.817276 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.817170    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6" event={"ID":"fe6940b7-0ad6-4a76-a69c-8595622d4e1e","Type":"ContainerDied","Data":"158800370eaa422c8a944b5046a4d7f779915b8a6180f24c188476ca57b05717"}
Sep 10 05:03:10.817276 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.817185    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6"
Sep 10 05:03:10.817276 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.817205    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6" event={"ID":"fe6940b7-0ad6-4a76-a69c-8595622d4e1e","Type":"ContainerDied","Data":"bcf476acdfaa5ec2e03a7b6602c17d01e52bfa7e8e2c51bac27b59d9c8da3681"}
Sep 10 05:03:10.826373 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.826318    2575 scope.go:117] "RemoveContainer" containerID="edf17ca6d033fb20b65c5f8b6f6137d2d24bd9f6fd4be59501bea8654eec9911"
Sep 10 05:03:10.826848 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:03:10.826704    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edf17ca6d033fb20b65c5f8b6f6137d2d24bd9f6fd4be59501bea8654eec9911\": container with ID starting with edf17ca6d033fb20b65c5f8b6f6137d2d24bd9f6fd4be59501bea8654eec9911 not found: ID does not exist" containerID="edf17ca6d033fb20b65c5f8b6f6137d2d24bd9f6fd4be59501bea8654eec9911"
Sep 10 05:03:10.826848 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.826741    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edf17ca6d033fb20b65c5f8b6f6137d2d24bd9f6fd4be59501bea8654eec9911"} err="failed to get container status \"edf17ca6d033fb20b65c5f8b6f6137d2d24bd9f6fd4be59501bea8654eec9911\": rpc error: code = NotFound desc = could not find container \"edf17ca6d033fb20b65c5f8b6f6137d2d24bd9f6fd4be59501bea8654eec9911\": container with ID starting with edf17ca6d033fb20b65c5f8b6f6137d2d24bd9f6fd4be59501bea8654eec9911 not found: ID does not exist"
Sep 10 05:03:10.826848 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.826764    2575 scope.go:117] "RemoveContainer" containerID="158800370eaa422c8a944b5046a4d7f779915b8a6180f24c188476ca57b05717"
Sep 10 05:03:10.835708 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.835689    2575 scope.go:117] "RemoveContainer" containerID="158800370eaa422c8a944b5046a4d7f779915b8a6180f24c188476ca57b05717"
Sep 10 05:03:10.835988 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:03:10.835969    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"158800370eaa422c8a944b5046a4d7f779915b8a6180f24c188476ca57b05717\": container with ID starting with 158800370eaa422c8a944b5046a4d7f779915b8a6180f24c188476ca57b05717 not found: ID does not exist" containerID="158800370eaa422c8a944b5046a4d7f779915b8a6180f24c188476ca57b05717"
Sep 10 05:03:10.836030 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.835995    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"158800370eaa422c8a944b5046a4d7f779915b8a6180f24c188476ca57b05717"} err="failed to get container status \"158800370eaa422c8a944b5046a4d7f779915b8a6180f24c188476ca57b05717\": rpc error: code = NotFound desc = could not find container \"158800370eaa422c8a944b5046a4d7f779915b8a6180f24c188476ca57b05717\": container with ID starting with 158800370eaa422c8a944b5046a4d7f779915b8a6180f24c188476ca57b05717 not found: ID does not exist"
Sep 10 05:03:10.838398 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.838376    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj"]
Sep 10 05:03:10.842142 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.842121    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8fcf3f8b-7dbfcbcc74-rxtzj"]
Sep 10 05:03:10.851821 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.851801    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6"]
Sep 10 05:03:10.858726 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:10.858700    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8fcf3f8b-7cd7cc74d9-w6nh6"]
Sep 10 05:03:11.863341 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:11.863297    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ba6690f-275e-47c4-addd-d164d36aba08" path="/var/lib/kubelet/pods/2ba6690f-275e-47c4-addd-d164d36aba08/volumes"
Sep 10 05:03:11.863823 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:11.863790    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2f494fa-d93c-42a2-bb80-de30c0642102" path="/var/lib/kubelet/pods/f2f494fa-d93c-42a2-bb80-de30c0642102/volumes"
Sep 10 05:03:11.864203 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:11.864184    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe6940b7-0ad6-4a76-a69c-8595622d4e1e" path="/var/lib/kubelet/pods/fe6940b7-0ad6-4a76-a69c-8595622d4e1e/volumes"
Sep 10 05:03:13.331018 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:13.330980    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:03:18.309293 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:18.309260    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"]
Sep 10 05:03:18.353901 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:18.353857    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:03:28.256353 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:28.256249    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:03:33.329485 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.329441    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms" podUID="604d2850-95e4-4039-b9a9-37f612c4aac1" containerName="main" containerID="cri-o://cfa30b9ff8345dbedbb368399f012139e36a2ec2129e6f973d7e0988e3bdd55f" gracePeriod=45
Sep 10 05:03:33.579234 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.579211    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:03:33.698223 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.698196    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97zlk\" (UniqueName: \"kubernetes.io/projected/604d2850-95e4-4039-b9a9-37f612c4aac1-kube-api-access-97zlk\") pod \"604d2850-95e4-4039-b9a9-37f612c4aac1\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") "
Sep 10 05:03:33.698370 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.698233    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-dshm\") pod \"604d2850-95e4-4039-b9a9-37f612c4aac1\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") "
Sep 10 05:03:33.698370 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.698286    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-model-cache\") pod \"604d2850-95e4-4039-b9a9-37f612c4aac1\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") "
Sep 10 05:03:33.698370 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.698367    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-home\") pod \"604d2850-95e4-4039-b9a9-37f612c4aac1\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") "
Sep 10 05:03:33.698507 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.698400    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-tmp-dir\") pod \"604d2850-95e4-4039-b9a9-37f612c4aac1\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") "
Sep 10 05:03:33.698507 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.698432    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/604d2850-95e4-4039-b9a9-37f612c4aac1-tls-certs\") pod \"604d2850-95e4-4039-b9a9-37f612c4aac1\" (UID: \"604d2850-95e4-4039-b9a9-37f612c4aac1\") "
Sep 10 05:03:33.698684 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.698630    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "604d2850-95e4-4039-b9a9-37f612c4aac1" (UID: "604d2850-95e4-4039-b9a9-37f612c4aac1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:03:33.698684 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.698658    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-model-cache" (OuterVolumeSpecName: "model-cache") pod "604d2850-95e4-4039-b9a9-37f612c4aac1" (UID: "604d2850-95e4-4039-b9a9-37f612c4aac1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:03:33.698684 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.698671    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-home" (OuterVolumeSpecName: "home") pod "604d2850-95e4-4039-b9a9-37f612c4aac1" (UID: "604d2850-95e4-4039-b9a9-37f612c4aac1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:03:33.698902 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.698733    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-home\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:03:33.698902 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.698749    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-tmp-dir\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:03:33.698902 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.698762    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-model-cache\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:03:33.700467 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.700437    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604d2850-95e4-4039-b9a9-37f612c4aac1-kube-api-access-97zlk" (OuterVolumeSpecName: "kube-api-access-97zlk") pod "604d2850-95e4-4039-b9a9-37f612c4aac1" (UID: "604d2850-95e4-4039-b9a9-37f612c4aac1"). InnerVolumeSpecName "kube-api-access-97zlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:03:33.700565 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.700447    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-dshm" (OuterVolumeSpecName: "dshm") pod "604d2850-95e4-4039-b9a9-37f612c4aac1" (UID: "604d2850-95e4-4039-b9a9-37f612c4aac1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:03:33.700565 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.700494    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604d2850-95e4-4039-b9a9-37f612c4aac1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "604d2850-95e4-4039-b9a9-37f612c4aac1" (UID: "604d2850-95e4-4039-b9a9-37f612c4aac1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 05:03:33.799991 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.799930    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/604d2850-95e4-4039-b9a9-37f612c4aac1-tls-certs\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:03:33.799991 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.799982    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97zlk\" (UniqueName: \"kubernetes.io/projected/604d2850-95e4-4039-b9a9-37f612c4aac1-kube-api-access-97zlk\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:03:33.799991 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.799996    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/604d2850-95e4-4039-b9a9-37f612c4aac1-dshm\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:03:33.934664 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.934623    2575 generic.go:356] "Generic (PLEG): container finished" podID="604d2850-95e4-4039-b9a9-37f612c4aac1" containerID="cfa30b9ff8345dbedbb368399f012139e36a2ec2129e6f973d7e0988e3bdd55f" exitCode=0
Sep 10 05:03:33.934664 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.934662    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"
Sep 10 05:03:33.934868 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.934705    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms" event={"ID":"604d2850-95e4-4039-b9a9-37f612c4aac1","Type":"ContainerDied","Data":"cfa30b9ff8345dbedbb368399f012139e36a2ec2129e6f973d7e0988e3bdd55f"}
Sep 10 05:03:33.934868 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.934748    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms" event={"ID":"604d2850-95e4-4039-b9a9-37f612c4aac1","Type":"ContainerDied","Data":"eef1eff5b5797eedfc0bdceb3fc54da87efd3538afbe8895cfef5a7fc3c85fd8"}
Sep 10 05:03:33.934868 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.934766    2575 scope.go:117] "RemoveContainer" containerID="cfa30b9ff8345dbedbb368399f012139e36a2ec2129e6f973d7e0988e3bdd55f"
Sep 10 05:03:33.943877 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.943859    2575 scope.go:117] "RemoveContainer" containerID="cfa30b9ff8345dbedbb368399f012139e36a2ec2129e6f973d7e0988e3bdd55f"
Sep 10 05:03:33.944217 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:03:33.944197    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfa30b9ff8345dbedbb368399f012139e36a2ec2129e6f973d7e0988e3bdd55f\": container with ID starting with cfa30b9ff8345dbedbb368399f012139e36a2ec2129e6f973d7e0988e3bdd55f not found: ID does not exist" containerID="cfa30b9ff8345dbedbb368399f012139e36a2ec2129e6f973d7e0988e3bdd55f"
Sep 10 05:03:33.944278 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.944225    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfa30b9ff8345dbedbb368399f012139e36a2ec2129e6f973d7e0988e3bdd55f"} err="failed to get container status \"cfa30b9ff8345dbedbb368399f012139e36a2ec2129e6f973d7e0988e3bdd55f\": rpc error: code = NotFound desc = could not find container \"cfa30b9ff8345dbedbb368399f012139e36a2ec2129e6f973d7e0988e3bdd55f\": container with ID starting with cfa30b9ff8345dbedbb368399f012139e36a2ec2129e6f973d7e0988e3bdd55f not found: ID does not exist"
Sep 10 05:03:33.949016 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.948950    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"]
Sep 10 05:03:33.951826 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:33.951802    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-8fcf3f8b/e2e-lifecycle-model-8fcf3f8b-kserve-64fc64f76c-fk6ms"]
Sep 10 05:03:35.863178 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:35.863145    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="604d2850-95e4-4039-b9a9-37f612c4aac1" path="/var/lib/kubelet/pods/604d2850-95e4-4039-b9a9-37f612c4aac1/volumes"
Sep 10 05:03:54.877499 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:54.877464    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:03:55.545785 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:03:55.545738    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:04.013721 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:04.013684    2575 scope.go:117] "RemoveContainer" containerID="e39d621bb4e49ffe6d944a373ec22370a52a4000b07cbbb5f5575819763ca919"
Sep 10 05:04:04.021818 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:04.021800    2575 scope.go:117] "RemoveContainer" containerID="e819599c4fe63f742ea85741b46f1c4eee4d878d4cf5ec09bb0834d0095cd655"
Sep 10 05:04:09.726728 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:09.726680    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:09.794811 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:09.794776    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:10.057869 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:10.057783    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:11.227055 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:11.227015    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:11.323668 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:11.323637    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:11.423469 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:11.423435    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:12.067533 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:12.067488    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:12.167390 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:12.167354    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:12.269366 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:12.269331    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:13.601041 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:13.600998    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:19.412146 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:19.412103    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:19.529175 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:19.529139    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:19.894001 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:19.893965    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:25.684485 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:25.684443    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:25.785645 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:25.785609    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:25.883587 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:25.883554    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:36.384205 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:36.384169    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:36.610513 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:36.610480    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:44.045297 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:44.045263    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"]
Sep 10 05:04:44.075527 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:44.075498    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:44.211178 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:44.211138    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:44.310547 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:44.310461    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:45.013011 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:45.012968    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"]
Sep 10 05:04:45.032862 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:45.032835    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:48.397962 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.397925    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"]
Sep 10 05:04:48.398358 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.398310    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2f494fa-d93c-42a2-bb80-de30c0642102" containerName="revoke-keys"
Sep 10 05:04:48.398358 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.398322    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2f494fa-d93c-42a2-bb80-de30c0642102" containerName="revoke-keys"
Sep 10 05:04:48.398358 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.398332    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ba6690f-275e-47c4-addd-d164d36aba08" containerName="payload-pre-processing"
Sep 10 05:04:48.398358 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.398339    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ba6690f-275e-47c4-addd-d164d36aba08" containerName="payload-pre-processing"
Sep 10 05:04:48.398358 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.398350    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="604d2850-95e4-4039-b9a9-37f612c4aac1" containerName="main"
Sep 10 05:04:48.398358 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.398358    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="604d2850-95e4-4039-b9a9-37f612c4aac1" containerName="main"
Sep 10 05:04:48.398550 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.398369    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe6940b7-0ad6-4a76-a69c-8595622d4e1e" containerName="payload-processing"
Sep 10 05:04:48.398550 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.398374    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe6940b7-0ad6-4a76-a69c-8595622d4e1e" containerName="payload-processing"
Sep 10 05:04:48.398550 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.398432    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2f494fa-d93c-42a2-bb80-de30c0642102" containerName="revoke-keys"
Sep 10 05:04:48.398550 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.398440    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="604d2850-95e4-4039-b9a9-37f612c4aac1" containerName="main"
Sep 10 05:04:48.398550 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.398445    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ba6690f-275e-47c4-addd-d164d36aba08" containerName="payload-pre-processing"
Sep 10 05:04:48.398550 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.398452    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe6940b7-0ad6-4a76-a69c-8595622d4e1e" containerName="payload-processing"
Sep 10 05:04:48.401322 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.401305    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:48.403573 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.403545    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4bfef6\"/\"sub-test-model-4bfef6-kserve-self-signed-certs\""
Sep 10 05:04:48.409391 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.409363    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"]
Sep 10 05:04:48.472074 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.472026    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-tmp-dir\") pod \"sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:48.472245 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.472089    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56edcfeb-2bfa-4b0b-b243-e623697ac986-tls-certs\") pod \"sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:48.472245 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.472188    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-dshm\") pod \"sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:48.472245 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.472224    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-model-cache\") pod \"sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:48.472416 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.472294    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6zph\" (UniqueName: \"kubernetes.io/projected/56edcfeb-2bfa-4b0b-b243-e623697ac986-kube-api-access-h6zph\") pod \"sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:48.472416 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.472332    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-home\") pod \"sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:48.499675 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.499637    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:48.573731 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.573693    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-dshm\") pod \"sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:48.573731 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.573733    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-model-cache\") pod \"sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:48.573965 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.573787    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6zph\" (UniqueName: \"kubernetes.io/projected/56edcfeb-2bfa-4b0b-b243-e623697ac986-kube-api-access-h6zph\") pod \"sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:48.573965 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.573816    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-home\") pod \"sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:48.573965 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.573845    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-tmp-dir\") pod \"sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:48.574161 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.574000    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56edcfeb-2bfa-4b0b-b243-e623697ac986-tls-certs\") pod \"sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:48.574232 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.574212    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-model-cache\") pod \"sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:48.574280 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.574229    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-tmp-dir\") pod \"sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:48.574320 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.574303    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-home\") pod \"sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:48.575910 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.575891    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-dshm\") pod \"sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:48.576422 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.576404    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56edcfeb-2bfa-4b0b-b243-e623697ac986-tls-certs\") pod \"sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:48.581430 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.581409    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6zph\" (UniqueName: \"kubernetes.io/projected/56edcfeb-2bfa-4b0b-b243-e623697ac986-kube-api-access-h6zph\") pod \"sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:48.710983 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.710954    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:48.985080 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:48.984963    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"]
Sep 10 05:04:48.987717 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:04:48.987675    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56edcfeb_2bfa_4b0b_b243_e623697ac986.slice/crio-0cdf2fbafb5a0af69ceb58023ff6103fbff2974b59f5e9069eecb18352a95f49 WatchSource:0}: Error finding container 0cdf2fbafb5a0af69ceb58023ff6103fbff2974b59f5e9069eecb18352a95f49: Status 404 returned error can't find the container with id 0cdf2fbafb5a0af69ceb58023ff6103fbff2974b59f5e9069eecb18352a95f49
Sep 10 05:04:49.303124 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:49.303008    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54" event={"ID":"56edcfeb-2bfa-4b0b-b243-e623697ac986","Type":"ContainerStarted","Data":"d55f2ac072dd40e9bda7a1ee9b9bd9ab99a0259a840c1c4f1cd74da4b57404e8"}
Sep 10 05:04:49.303124 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:49.303087    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54" event={"ID":"56edcfeb-2bfa-4b0b-b243-e623697ac986","Type":"ContainerStarted","Data":"0cdf2fbafb5a0af69ceb58023ff6103fbff2974b59f5e9069eecb18352a95f49"}
Sep 10 05:04:49.318698 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:49.318645    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54" podStartSLOduration=1.318629649 podStartE2EDuration="1.318629649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:04:49.317596953 +0000 UTC m=+1426.079366136" watchObservedRunningTime="2026-09-10 05:04:49.318629649 +0000 UTC m=+1426.080398823"
Sep 10 05:04:57.602728 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:57.602637    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:04:58.711499 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:58.711467    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:58.711886 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:58.711518    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:58.724189 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:58.724158    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:59.064959 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.064854    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h" podUID="cfbdc46d-97c5-4a4c-9fef-0577fd6adc63" containerName="main" containerID="cri-o://086d1ac69b8a1df0433c5c407e2a8f34a0d10ecc9eb13db98f410abc92491add" gracePeriod=45
Sep 10 05:04:59.308538 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.308514    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:04:59.353313 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.353228    2575 generic.go:356] "Generic (PLEG): container finished" podID="cfbdc46d-97c5-4a4c-9fef-0577fd6adc63" containerID="086d1ac69b8a1df0433c5c407e2a8f34a0d10ecc9eb13db98f410abc92491add" exitCode=0
Sep 10 05:04:59.353313 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.353263    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"
Sep 10 05:04:59.353313 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.353308    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h" event={"ID":"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63","Type":"ContainerDied","Data":"086d1ac69b8a1df0433c5c407e2a8f34a0d10ecc9eb13db98f410abc92491add"}
Sep 10 05:04:59.353633 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.353348    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h" event={"ID":"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63","Type":"ContainerDied","Data":"0ed206883728dc12cd929897f09a58bf2ee40d889a13e35d6bb2c19df4e280b6"}
Sep 10 05:04:59.353633 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.353370    2575 scope.go:117] "RemoveContainer" containerID="086d1ac69b8a1df0433c5c407e2a8f34a0d10ecc9eb13db98f410abc92491add"
Sep 10 05:04:59.362035 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.362017    2575 scope.go:117] "RemoveContainer" containerID="086d1ac69b8a1df0433c5c407e2a8f34a0d10ecc9eb13db98f410abc92491add"
Sep 10 05:04:59.362330 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:04:59.362313    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"086d1ac69b8a1df0433c5c407e2a8f34a0d10ecc9eb13db98f410abc92491add\": container with ID starting with 086d1ac69b8a1df0433c5c407e2a8f34a0d10ecc9eb13db98f410abc92491add not found: ID does not exist" containerID="086d1ac69b8a1df0433c5c407e2a8f34a0d10ecc9eb13db98f410abc92491add"
Sep 10 05:04:59.362397 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.362338    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"086d1ac69b8a1df0433c5c407e2a8f34a0d10ecc9eb13db98f410abc92491add"} err="failed to get container status \"086d1ac69b8a1df0433c5c407e2a8f34a0d10ecc9eb13db98f410abc92491add\": rpc error: code = NotFound desc = could not find container \"086d1ac69b8a1df0433c5c407e2a8f34a0d10ecc9eb13db98f410abc92491add\": container with ID starting with 086d1ac69b8a1df0433c5c407e2a8f34a0d10ecc9eb13db98f410abc92491add not found: ID does not exist"
Sep 10 05:04:59.365432 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.365411    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:04:59.472446 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.472412    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-tls-certs\") pod \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") "
Sep 10 05:04:59.472618 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.472513    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-model-cache\") pod \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") "
Sep 10 05:04:59.472618 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.472534    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-dshm\") pod \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") "
Sep 10 05:04:59.472618 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.472583    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2g4q\" (UniqueName: \"kubernetes.io/projected/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-kube-api-access-f2g4q\") pod \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") "
Sep 10 05:04:59.472618 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.472615    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-tmp-dir\") pod \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") "
Sep 10 05:04:59.472834 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.472637    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-home\") pod \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\" (UID: \"cfbdc46d-97c5-4a4c-9fef-0577fd6adc63\") "
Sep 10 05:04:59.472887 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.472821    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfbdc46d-97c5-4a4c-9fef-0577fd6adc63" (UID: "cfbdc46d-97c5-4a4c-9fef-0577fd6adc63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:04:59.472887 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.472856    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cfbdc46d-97c5-4a4c-9fef-0577fd6adc63" (UID: "cfbdc46d-97c5-4a4c-9fef-0577fd6adc63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:04:59.473005 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.472946    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-home" (OuterVolumeSpecName: "home") pod "cfbdc46d-97c5-4a4c-9fef-0577fd6adc63" (UID: "cfbdc46d-97c5-4a4c-9fef-0577fd6adc63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:04:59.473106 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.473013    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-tmp-dir\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:04:59.473106 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.473032    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-model-cache\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:04:59.474880 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.474856    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-kube-api-access-f2g4q" (OuterVolumeSpecName: "kube-api-access-f2g4q") pod "cfbdc46d-97c5-4a4c-9fef-0577fd6adc63" (UID: "cfbdc46d-97c5-4a4c-9fef-0577fd6adc63"). InnerVolumeSpecName "kube-api-access-f2g4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:04:59.475040 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.475026    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cfbdc46d-97c5-4a4c-9fef-0577fd6adc63" (UID: "cfbdc46d-97c5-4a4c-9fef-0577fd6adc63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 05:04:59.475106 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.475056    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-dshm" (OuterVolumeSpecName: "dshm") pod "cfbdc46d-97c5-4a4c-9fef-0577fd6adc63" (UID: "cfbdc46d-97c5-4a4c-9fef-0577fd6adc63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:04:59.574337 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.574306    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-dshm\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:04:59.574337 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.574334    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2g4q\" (UniqueName: \"kubernetes.io/projected/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-kube-api-access-f2g4q\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:04:59.574337 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.574345    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-home\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:04:59.574559 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.574354    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63-tls-certs\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:04:59.677410 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.677380    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"]
Sep 10 05:04:59.682687 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.682663    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4bfef6/auth-test-model-4bfef6-kserve-58dddcdf47-2cn5h"]
Sep 10 05:04:59.863252 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:04:59.863221    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfbdc46d-97c5-4a4c-9fef-0577fd6adc63" path="/var/lib/kubelet/pods/cfbdc46d-97c5-4a4c-9fef-0577fd6adc63/volumes"
Sep 10 05:05:00.032340 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.032278    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v" podUID="5508ef62-7e0c-4d0d-8d16-84b55aa30bf6" containerName="main" containerID="cri-o://b6f650a1124ee1ad847b2b6b3762cf76d16911a369055f0dd8dc2c3ae0d6d29c" gracePeriod=45
Sep 10 05:05:00.279828 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.279803    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:05:00.359819 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.359729    2575 generic.go:356] "Generic (PLEG): container finished" podID="5508ef62-7e0c-4d0d-8d16-84b55aa30bf6" containerID="b6f650a1124ee1ad847b2b6b3762cf76d16911a369055f0dd8dc2c3ae0d6d29c" exitCode=0
Sep 10 05:05:00.359819 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.359773    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"
Sep 10 05:05:00.360018 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.359826    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v" event={"ID":"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6","Type":"ContainerDied","Data":"b6f650a1124ee1ad847b2b6b3762cf76d16911a369055f0dd8dc2c3ae0d6d29c"}
Sep 10 05:05:00.360018 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.359867    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v" event={"ID":"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6","Type":"ContainerDied","Data":"0919259f115712e6379171a03d16d4dfd44d65000da4bca99de4166d371b4f62"}
Sep 10 05:05:00.360018 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.359892    2575 scope.go:117] "RemoveContainer" containerID="b6f650a1124ee1ad847b2b6b3762cf76d16911a369055f0dd8dc2c3ae0d6d29c"
Sep 10 05:05:00.368215 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.368198    2575 scope.go:117] "RemoveContainer" containerID="b6f650a1124ee1ad847b2b6b3762cf76d16911a369055f0dd8dc2c3ae0d6d29c"
Sep 10 05:05:00.368483 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:05:00.368465    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6f650a1124ee1ad847b2b6b3762cf76d16911a369055f0dd8dc2c3ae0d6d29c\": container with ID starting with b6f650a1124ee1ad847b2b6b3762cf76d16911a369055f0dd8dc2c3ae0d6d29c not found: ID does not exist" containerID="b6f650a1124ee1ad847b2b6b3762cf76d16911a369055f0dd8dc2c3ae0d6d29c"
Sep 10 05:05:00.368526 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.368493    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6f650a1124ee1ad847b2b6b3762cf76d16911a369055f0dd8dc2c3ae0d6d29c"} err="failed to get container status \"b6f650a1124ee1ad847b2b6b3762cf76d16911a369055f0dd8dc2c3ae0d6d29c\": rpc error: code = NotFound desc = could not find container \"b6f650a1124ee1ad847b2b6b3762cf76d16911a369055f0dd8dc2c3ae0d6d29c\": container with ID starting with b6f650a1124ee1ad847b2b6b3762cf76d16911a369055f0dd8dc2c3ae0d6d29c not found: ID does not exist"
Sep 10 05:05:00.380768 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.380747    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-model-cache\") pod \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") "
Sep 10 05:05:00.380839 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.380778    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xr2g\" (UniqueName: \"kubernetes.io/projected/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-kube-api-access-6xr2g\") pod \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") "
Sep 10 05:05:00.380876 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.380863    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-tls-certs\") pod \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") "
Sep 10 05:05:00.380918 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.380893    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-dshm\") pod \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") "
Sep 10 05:05:00.380959 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.380937    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-home\") pod \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") "
Sep 10 05:05:00.381015 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.380972    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-tmp-dir\") pod \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\" (UID: \"5508ef62-7e0c-4d0d-8d16-84b55aa30bf6\") "
Sep 10 05:05:00.381015 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.380980    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-model-cache" (OuterVolumeSpecName: "model-cache") pod "5508ef62-7e0c-4d0d-8d16-84b55aa30bf6" (UID: "5508ef62-7e0c-4d0d-8d16-84b55aa30bf6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:05:00.381287 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.381247    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-home" (OuterVolumeSpecName: "home") pod "5508ef62-7e0c-4d0d-8d16-84b55aa30bf6" (UID: "5508ef62-7e0c-4d0d-8d16-84b55aa30bf6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:05:00.381407 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.381278    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5508ef62-7e0c-4d0d-8d16-84b55aa30bf6" (UID: "5508ef62-7e0c-4d0d-8d16-84b55aa30bf6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:05:00.381407 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.381338    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-model-cache\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:05:00.383061 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.383028    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-dshm" (OuterVolumeSpecName: "dshm") pod "5508ef62-7e0c-4d0d-8d16-84b55aa30bf6" (UID: "5508ef62-7e0c-4d0d-8d16-84b55aa30bf6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:05:00.383166 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.383129    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-kube-api-access-6xr2g" (OuterVolumeSpecName: "kube-api-access-6xr2g") pod "5508ef62-7e0c-4d0d-8d16-84b55aa30bf6" (UID: "5508ef62-7e0c-4d0d-8d16-84b55aa30bf6"). InnerVolumeSpecName "kube-api-access-6xr2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:05:00.383260 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.383242    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5508ef62-7e0c-4d0d-8d16-84b55aa30bf6" (UID: "5508ef62-7e0c-4d0d-8d16-84b55aa30bf6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 05:05:00.481974 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.481935    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xr2g\" (UniqueName: \"kubernetes.io/projected/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-kube-api-access-6xr2g\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:05:00.481974 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.481966    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-tls-certs\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:05:00.481974 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.481976    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-dshm\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:05:00.481974 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.481984    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-home\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:05:00.482295 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.481992    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6-tmp-dir\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:05:00.680340 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.680310    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"]
Sep 10 05:05:00.683813 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:00.683789    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b702d/auth-test-model-3b702d-kserve-55c9f695f5-rnl9v"]
Sep 10 05:05:01.863372 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:01.863328    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5508ef62-7e0c-4d0d-8d16-84b55aa30bf6" path="/var/lib/kubelet/pods/5508ef62-7e0c-4d0d-8d16-84b55aa30bf6/volumes"
Sep 10 05:05:03.533037 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:03.532971    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:03.595556 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:03.595519    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:03.908263 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:03.908178    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:04.229559 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:04.229517    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:04.603377 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:04.603272    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:07.114404 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:07.114361    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:07.201693 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:07.201661    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:08.436774 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:08.436737    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:08.503086 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:08.503032    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:08.620875 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:08.620839    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:08.882256 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:08.882214    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:08.991749 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:08.991704    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:32.132096 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:32.132043    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:32.482150 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:32.482101    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:32.537886 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:32.537856    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:32.625705 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:32.625669    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:32.947323 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:32.947283    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:33.048292 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:33.048254    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:37.115714 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:37.115674    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:37.215943 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:37.215905    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:37.316436 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:37.316408    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:37.416356 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:37.416249    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:37.741969 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:37.741930    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:37.836820 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:37.836788    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h"]
Sep 10 05:05:37.837161 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:37.837147    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5508ef62-7e0c-4d0d-8d16-84b55aa30bf6" containerName="main"
Sep 10 05:05:37.837219 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:37.837163    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5508ef62-7e0c-4d0d-8d16-84b55aa30bf6" containerName="main"
Sep 10 05:05:37.837219 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:37.837176    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfbdc46d-97c5-4a4c-9fef-0577fd6adc63" containerName="main"
Sep 10 05:05:37.837219 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:37.837182    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfbdc46d-97c5-4a4c-9fef-0577fd6adc63" containerName="main"
Sep 10 05:05:37.837318 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:37.837244    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfbdc46d-97c5-4a4c-9fef-0577fd6adc63" containerName="main"
Sep 10 05:05:37.837318 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:37.837254    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5508ef62-7e0c-4d0d-8d16-84b55aa30bf6" containerName="main"
Sep 10 05:05:37.840338 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:37.840313    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h"
Sep 10 05:05:37.840478 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:37.840380    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:37.842882 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:37.842866    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-25a681-dockercfg-nv9pl\""
Sep 10 05:05:37.842986 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:37.842867    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-25a681\""
Sep 10 05:05:37.852854 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:37.852822    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h"]
Sep 10 05:05:37.868376 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:37.868349    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72"]
Sep 10 05:05:37.871785 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:37.871763    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72"
Sep 10 05:05:37.874382 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:37.874359    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72"]
Sep 10 05:05:37.920801 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:37.920764    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h\" (UID: \"f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h"
Sep 10 05:05:37.920959 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:37.920834    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jfqn\" (UniqueName: \"kubernetes.io/projected/f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5-kube-api-access-9jfqn\") pod \"payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h\" (UID: \"f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h"
Sep 10 05:05:38.021905 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:38.021823    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5efb1652-c513-47f7-be66-51c2bb329578-plugins-config-volume\") pod \"payload-processing-e2e-api-b-25a681-589db85f95-hgg72\" (UID: \"5efb1652-c513-47f7-be66-51c2bb329578\") " pod="openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72"
Sep 10 05:05:38.021905 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:38.021869    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwb9t\" (UniqueName: \"kubernetes.io/projected/5efb1652-c513-47f7-be66-51c2bb329578-kube-api-access-hwb9t\") pod \"payload-processing-e2e-api-b-25a681-589db85f95-hgg72\" (UID: \"5efb1652-c513-47f7-be66-51c2bb329578\") " pod="openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72"
Sep 10 05:05:38.021905 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:38.021904    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h\" (UID: \"f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h"
Sep 10 05:05:38.022148 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:38.021947    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jfqn\" (UniqueName: \"kubernetes.io/projected/f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5-kube-api-access-9jfqn\") pod \"payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h\" (UID: \"f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h"
Sep 10 05:05:38.022617 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:38.022597    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h\" (UID: \"f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h"
Sep 10 05:05:38.033444 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:38.033416    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jfqn\" (UniqueName: \"kubernetes.io/projected/f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5-kube-api-access-9jfqn\") pod \"payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h\" (UID: \"f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h"
Sep 10 05:05:38.123118 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:38.123060    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5efb1652-c513-47f7-be66-51c2bb329578-plugins-config-volume\") pod \"payload-processing-e2e-api-b-25a681-589db85f95-hgg72\" (UID: \"5efb1652-c513-47f7-be66-51c2bb329578\") " pod="openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72"
Sep 10 05:05:38.123582 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:38.123127    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwb9t\" (UniqueName: \"kubernetes.io/projected/5efb1652-c513-47f7-be66-51c2bb329578-kube-api-access-hwb9t\") pod \"payload-processing-e2e-api-b-25a681-589db85f95-hgg72\" (UID: \"5efb1652-c513-47f7-be66-51c2bb329578\") " pod="openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72"
Sep 10 05:05:38.123891 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:38.123867    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5efb1652-c513-47f7-be66-51c2bb329578-plugins-config-volume\") pod \"payload-processing-e2e-api-b-25a681-589db85f95-hgg72\" (UID: \"5efb1652-c513-47f7-be66-51c2bb329578\") " pod="openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72"
Sep 10 05:05:38.131226 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:38.131201    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwb9t\" (UniqueName: \"kubernetes.io/projected/5efb1652-c513-47f7-be66-51c2bb329578-kube-api-access-hwb9t\") pod \"payload-processing-e2e-api-b-25a681-589db85f95-hgg72\" (UID: \"5efb1652-c513-47f7-be66-51c2bb329578\") " pod="openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72"
Sep 10 05:05:38.151899 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:38.151875    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h"
Sep 10 05:05:38.183177 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:38.183151    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72"
Sep 10 05:05:38.423000 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:38.422970    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h"]
Sep 10 05:05:38.425048 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:05:38.425017    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8cc1fdf_ec26_48fb_9596_24c2a24fc5e5.slice/crio-db39d374586a0199c761be778fb75231befc6b2c0640c4417af45463409be873 WatchSource:0}: Error finding container db39d374586a0199c761be778fb75231befc6b2c0640c4417af45463409be873: Status 404 returned error can't find the container with id db39d374586a0199c761be778fb75231befc6b2c0640c4417af45463409be873
Sep 10 05:05:38.463687 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:38.463664    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72"]
Sep 10 05:05:38.466338 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:05:38.466314    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5efb1652_c513_47f7_be66_51c2bb329578.slice/crio-5915fa142d17ee2747dfd18cbc63afd51d667bd12362d2efcdd03ca34f102aa2 WatchSource:0}: Error finding container 5915fa142d17ee2747dfd18cbc63afd51d667bd12362d2efcdd03ca34f102aa2: Status 404 returned error can't find the container with id 5915fa142d17ee2747dfd18cbc63afd51d667bd12362d2efcdd03ca34f102aa2
Sep 10 05:05:38.561374 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:38.561317    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72" event={"ID":"5efb1652-c513-47f7-be66-51c2bb329578","Type":"ContainerStarted","Data":"cc5c2323b94797c596ac48fd675d976a443ef096c6de5d408ffe2bdd554e76e1"}
Sep 10 05:05:38.561786 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:38.561404    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72"
Sep 10 05:05:38.561786 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:38.561431    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72" event={"ID":"5efb1652-c513-47f7-be66-51c2bb329578","Type":"ContainerStarted","Data":"5915fa142d17ee2747dfd18cbc63afd51d667bd12362d2efcdd03ca34f102aa2"}
Sep 10 05:05:38.563096 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:38.563047    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h" event={"ID":"f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5","Type":"ContainerStarted","Data":"fff822e085f0e224dc4c7249cb576ad66a20fe0e0d3fadc896c97413c2c30ee9"}
Sep 10 05:05:38.563236 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:38.563114    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h" event={"ID":"f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5","Type":"ContainerStarted","Data":"db39d374586a0199c761be778fb75231befc6b2c0640c4417af45463409be873"}
Sep 10 05:05:38.563236 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:38.563161    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h"
Sep 10 05:05:38.579647 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:38.579579    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72" podStartSLOduration=1.579558435 podStartE2EDuration="1.579558435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:05:38.576059879 +0000 UTC m=+1475.337829062" watchObservedRunningTime="2026-09-10 05:05:38.579558435 +0000 UTC m=+1475.341327622"
Sep 10 05:05:38.592643 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:38.592587    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h" podStartSLOduration=1.59256738 podStartE2EDuration="1.59256738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:05:38.591330425 +0000 UTC m=+1475.353099607" watchObservedRunningTime="2026-09-10 05:05:38.59256738 +0000 UTC m=+1475.354336764"
Sep 10 05:05:39.043881 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:39.043829    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:39.832568 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:39.832534    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:45.453819 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:45.453783    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:46.190708 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.190669    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"]
Sep 10 05:05:46.194311 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.194289    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:46.197037 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.197002    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b702d\"/\"default-dockercfg-2ln7j\""
Sep 10 05:05:46.198030 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.198002    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b702d\"/\"openshift-service-ca.crt\""
Sep 10 05:05:46.198183 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.198170    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b702d\"/\"sub-test-model-3b702d-kserve-self-signed-certs\""
Sep 10 05:05:46.198285 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.198255    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b702d\"/\"kube-root-ca.crt\""
Sep 10 05:05:46.202552 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.202531    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"]
Sep 10 05:05:46.220297 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.220264    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:05:46.298438 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.298405    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-home\") pod \"sub-test-model-3b702d-kserve-57b84d486c-qblbh\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:46.298584 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.298445    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-dshm\") pod \"sub-test-model-3b702d-kserve-57b84d486c-qblbh\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:46.298584 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.298502    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-tls-certs\") pod \"sub-test-model-3b702d-kserve-57b84d486c-qblbh\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:46.298584 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.298567    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-tmp-dir\") pod \"sub-test-model-3b702d-kserve-57b84d486c-qblbh\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:46.298712 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.298601    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-model-cache\") pod \"sub-test-model-3b702d-kserve-57b84d486c-qblbh\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:46.298712 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.298662    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpdrv\" (UniqueName: \"kubernetes.io/projected/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-kube-api-access-tpdrv\") pod \"sub-test-model-3b702d-kserve-57b84d486c-qblbh\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:46.399771 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.399734    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-tmp-dir\") pod \"sub-test-model-3b702d-kserve-57b84d486c-qblbh\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:46.399771 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.399774    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-model-cache\") pod \"sub-test-model-3b702d-kserve-57b84d486c-qblbh\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:46.399961 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.399897    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpdrv\" (UniqueName: \"kubernetes.io/projected/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-kube-api-access-tpdrv\") pod \"sub-test-model-3b702d-kserve-57b84d486c-qblbh\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:46.400017 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.399978    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-home\") pod \"sub-test-model-3b702d-kserve-57b84d486c-qblbh\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:46.400017 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.399996    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-dshm\") pod \"sub-test-model-3b702d-kserve-57b84d486c-qblbh\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:46.400154 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.400019    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-tls-certs\") pod \"sub-test-model-3b702d-kserve-57b84d486c-qblbh\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:46.400206 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.400184    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-model-cache\") pod \"sub-test-model-3b702d-kserve-57b84d486c-qblbh\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:46.400258 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.400215    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-tmp-dir\") pod \"sub-test-model-3b702d-kserve-57b84d486c-qblbh\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:46.400303 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.400295    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-home\") pod \"sub-test-model-3b702d-kserve-57b84d486c-qblbh\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:46.402336 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.402314    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-dshm\") pod \"sub-test-model-3b702d-kserve-57b84d486c-qblbh\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:46.402492 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.402476    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-tls-certs\") pod \"sub-test-model-3b702d-kserve-57b84d486c-qblbh\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:46.408033 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.408010    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpdrv\" (UniqueName: \"kubernetes.io/projected/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-kube-api-access-tpdrv\") pod \"sub-test-model-3b702d-kserve-57b84d486c-qblbh\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:46.504913 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.504878    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:46.773773 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:46.773751    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"]
Sep 10 05:05:46.776518 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:05:46.776493    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod820e5f0d_7fcb_4ef4_adc6_0d62fa16491c.slice/crio-594a8c4efcfda6b32c1cd9ab0206bdb9d7754cbdaf664c3370edb6e427741632 WatchSource:0}: Error finding container 594a8c4efcfda6b32c1cd9ab0206bdb9d7754cbdaf664c3370edb6e427741632: Status 404 returned error can't find the container with id 594a8c4efcfda6b32c1cd9ab0206bdb9d7754cbdaf664c3370edb6e427741632
Sep 10 05:05:47.616177 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:47.616136    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh" event={"ID":"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c","Type":"ContainerStarted","Data":"81baf6c04a5207208e255b55f3f3aee42835fb9b75d95a58749ef98076e40384"}
Sep 10 05:05:47.616177 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:47.616180    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh" event={"ID":"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c","Type":"ContainerStarted","Data":"594a8c4efcfda6b32c1cd9ab0206bdb9d7754cbdaf664c3370edb6e427741632"}
Sep 10 05:05:47.632875 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:47.632802    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh" podStartSLOduration=1.632787282 podStartE2EDuration="1.632787282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:05:47.632471509 +0000 UTC m=+1484.394240691" watchObservedRunningTime="2026-09-10 05:05:47.632787282 +0000 UTC m=+1484.394556464"
Sep 10 05:05:49.571117 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:49.571086    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h"
Sep 10 05:05:49.571496 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:49.571157    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72"
Sep 10 05:05:56.505811 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:56.505767    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:56.506237 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:56.505843    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:56.518552 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:56.518522    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:05:56.677309 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:05:56.677276    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:06:01.091321 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:01.091284    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:03.775158 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:03.775126    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2hd2_9d909143-4c70-4fe1-900a-3cd520db32e5/ovn-acl-logging/0.log"
Sep 10 05:06:03.886056 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:03.886029    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2hd2_9d909143-4c70-4fe1-900a-3cd520db32e5/ovn-acl-logging/0.log"
Sep 10 05:06:04.067574 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:04.067476    2575 scope.go:117] "RemoveContainer" containerID="05336f31a7587f84c75a1208036ecaed4a590eb854cc77fed782eec1a2f36028"
Sep 10 05:06:04.075856 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:04.075837    2575 scope.go:117] "RemoveContainer" containerID="2139e10a6d70d18e855d4a00b4fd33f364623e65f52f58aabef6a93201cec917"
Sep 10 05:06:04.083855 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:04.083837    2575 scope.go:117] "RemoveContainer" containerID="15d56d8d2958cbe54654af08df863556dfe079b234586a4e103c42fdbfacd830"
Sep 10 05:06:04.090931 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:04.090916    2575 scope.go:117] "RemoveContainer" containerID="0c5a42098e02d22dbe8e463723154aa635652855b320c89e611b75950b6531e7"
Sep 10 05:06:04.171141 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:04.171108    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5e779-gkg86"]
Sep 10 05:06:04.176321 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:04.176298    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5e779-gkg86"
Sep 10 05:06:04.179010 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:04.178823    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 05:06:04.179615 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:04.179593    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dkf6w\""
Sep 10 05:06:04.181524 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:04.181501    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5e779-gkg86"]
Sep 10 05:06:04.228770 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:04.228733    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa12655-8e7f-4b74-8d83-7c467c71faeb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c5e779-gkg86\" (UID: \"aaa12655-8e7f-4b74-8d83-7c467c71faeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5e779-gkg86"
Sep 10 05:06:04.329593 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:04.329514    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa12655-8e7f-4b74-8d83-7c467c71faeb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c5e779-gkg86\" (UID: \"aaa12655-8e7f-4b74-8d83-7c467c71faeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5e779-gkg86"
Sep 10 05:06:04.330168 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:04.330147    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa12655-8e7f-4b74-8d83-7c467c71faeb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c5e779-gkg86\" (UID: \"aaa12655-8e7f-4b74-8d83-7c467c71faeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5e779-gkg86"
Sep 10 05:06:04.480756 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:04.480711    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:04.490371 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:04.490333    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5e779-gkg86"
Sep 10 05:06:04.760189 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:04.760164    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5e779-gkg86"]
Sep 10 05:06:04.762304 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:06:04.762280    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa12655_8e7f_4b74_8d83_7c467c71faeb.slice/crio-760d9b160f1d311dd4009e5ea3ab9545270e2546df15e85d9fb1fffb35094912 WatchSource:0}: Error finding container 760d9b160f1d311dd4009e5ea3ab9545270e2546df15e85d9fb1fffb35094912: Status 404 returned error can't find the container with id 760d9b160f1d311dd4009e5ea3ab9545270e2546df15e85d9fb1fffb35094912
Sep 10 05:06:05.720879 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:05.720848    2575 generic.go:356] "Generic (PLEG): container finished" podID="aaa12655-8e7f-4b74-8d83-7c467c71faeb" containerID="855c31d89a68b6294003780f40ce54538a6cf528fc9e6b91b8f1b05cdb47a27c" exitCode=0
Sep 10 05:06:05.721277 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:05.720929    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5e779-gkg86" event={"ID":"aaa12655-8e7f-4b74-8d83-7c467c71faeb","Type":"ContainerDied","Data":"855c31d89a68b6294003780f40ce54538a6cf528fc9e6b91b8f1b05cdb47a27c"}
Sep 10 05:06:05.721277 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:05.720978    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5e779-gkg86" event={"ID":"aaa12655-8e7f-4b74-8d83-7c467c71faeb","Type":"ContainerStarted","Data":"760d9b160f1d311dd4009e5ea3ab9545270e2546df15e85d9fb1fffb35094912"}
Sep 10 05:06:06.853707 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:06.853674    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5e779-gkg86"
Sep 10 05:06:06.954737 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:06.954698    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa12655-8e7f-4b74-8d83-7c467c71faeb-maas-api-service-ca\") pod \"aaa12655-8e7f-4b74-8d83-7c467c71faeb\" (UID: \"aaa12655-8e7f-4b74-8d83-7c467c71faeb\") "
Sep 10 05:06:06.955087 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:06.955042    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa12655-8e7f-4b74-8d83-7c467c71faeb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aaa12655-8e7f-4b74-8d83-7c467c71faeb" (UID: "aaa12655-8e7f-4b74-8d83-7c467c71faeb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:06:07.056392 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:07.056311    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa12655-8e7f-4b74-8d83-7c467c71faeb-maas-api-service-ca\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:06:07.731756 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:07.731727    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5e779-gkg86"
Sep 10 05:06:07.731958 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:07.731771    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5e779-gkg86" event={"ID":"aaa12655-8e7f-4b74-8d83-7c467c71faeb","Type":"ContainerDied","Data":"760d9b160f1d311dd4009e5ea3ab9545270e2546df15e85d9fb1fffb35094912"}
Sep 10 05:06:07.731958 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:07.731819    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="760d9b160f1d311dd4009e5ea3ab9545270e2546df15e85d9fb1fffb35094912"
Sep 10 05:06:08.256670 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:08.256627    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:11.040476 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:11.040428    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:14.317648 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:14.317609    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:19.876814 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:19.876779    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:19.942311 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:19.942266    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:20.041609 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:20.041545    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:20.057816 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:20.057778    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5e779-gkg86"]
Sep 10 05:06:20.059716 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:20.059698    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5e779-gkg86"]
Sep 10 05:06:20.117371 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:20.117337    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:20.467775 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:20.467735    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:21.863109 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:21.863056    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaa12655-8e7f-4b74-8d83-7c467c71faeb" path="/var/lib/kubelet/pods/aaa12655-8e7f-4b74-8d83-7c467c71faeb/volumes"
Sep 10 05:06:26.438489 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:26.438395    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:26.538814 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:26.538774    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:26.696950 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:26.696850    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:27.049134 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:27.049098    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25a681-pmq2z"]
Sep 10 05:06:27.049495 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:27.049483    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa12655-8e7f-4b74-8d83-7c467c71faeb" containerName="revoke-keys"
Sep 10 05:06:27.049562 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:27.049498    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa12655-8e7f-4b74-8d83-7c467c71faeb" containerName="revoke-keys"
Sep 10 05:06:27.049562 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:27.049556    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaa12655-8e7f-4b74-8d83-7c467c71faeb" containerName="revoke-keys"
Sep 10 05:06:27.053664 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:27.053645    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25a681-pmq2z"
Sep 10 05:06:27.055924 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:27.055901    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 05:06:27.056024 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:27.055950    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dkf6w\""
Sep 10 05:06:27.060330 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:27.060306    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25a681-pmq2z"]
Sep 10 05:06:27.136382 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:27.136346    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/209627bf-2a35-4b35-91d6-e3d818149f42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-25a681-pmq2z\" (UID: \"209627bf-2a35-4b35-91d6-e3d818149f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25a681-pmq2z"
Sep 10 05:06:27.161566 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:27.161527    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:27.237639 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:27.237587    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/209627bf-2a35-4b35-91d6-e3d818149f42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-25a681-pmq2z\" (UID: \"209627bf-2a35-4b35-91d6-e3d818149f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25a681-pmq2z"
Sep 10 05:06:27.238284 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:27.238263    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/209627bf-2a35-4b35-91d6-e3d818149f42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-25a681-pmq2z\" (UID: \"209627bf-2a35-4b35-91d6-e3d818149f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25a681-pmq2z"
Sep 10 05:06:27.365326 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:27.365234    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25a681-pmq2z"
Sep 10 05:06:27.636905 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:27.636878    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25a681-pmq2z"]
Sep 10 05:06:27.639336 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:06:27.639301    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209627bf_2a35_4b35_91d6_e3d818149f42.slice/crio-9ef9cc9a99c9db6d852c7defe4b5f045656ef25454da335a884ed37c13e53e58 WatchSource:0}: Error finding container 9ef9cc9a99c9db6d852c7defe4b5f045656ef25454da335a884ed37c13e53e58: Status 404 returned error can't find the container with id 9ef9cc9a99c9db6d852c7defe4b5f045656ef25454da335a884ed37c13e53e58
Sep 10 05:06:27.833840 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:27.833809    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25a681-pmq2z" event={"ID":"209627bf-2a35-4b35-91d6-e3d818149f42","Type":"ContainerStarted","Data":"9ef9cc9a99c9db6d852c7defe4b5f045656ef25454da335a884ed37c13e53e58"}
Sep 10 05:06:28.849361 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:28.849326    2575 generic.go:356] "Generic (PLEG): container finished" podID="209627bf-2a35-4b35-91d6-e3d818149f42" containerID="9da88d016c758f97bd7b939b3f9260d31b291edeb7881195a16b5e7b4f533c03" exitCode=0
Sep 10 05:06:28.849743 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:28.849407    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25a681-pmq2z" event={"ID":"209627bf-2a35-4b35-91d6-e3d818149f42","Type":"ContainerDied","Data":"9da88d016c758f97bd7b939b3f9260d31b291edeb7881195a16b5e7b4f533c03"}
Sep 10 05:06:29.986644 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:29.986615    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25a681-pmq2z"
Sep 10 05:06:30.063078 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:30.063037    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/209627bf-2a35-4b35-91d6-e3d818149f42-maas-api-service-ca\") pod \"209627bf-2a35-4b35-91d6-e3d818149f42\" (UID: \"209627bf-2a35-4b35-91d6-e3d818149f42\") "
Sep 10 05:06:30.063405 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:30.063381    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/209627bf-2a35-4b35-91d6-e3d818149f42-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "209627bf-2a35-4b35-91d6-e3d818149f42" (UID: "209627bf-2a35-4b35-91d6-e3d818149f42"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:06:30.164393 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:30.164299    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/209627bf-2a35-4b35-91d6-e3d818149f42-maas-api-service-ca\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:06:30.860863 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:30.860831    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25a681-pmq2z"
Sep 10 05:06:30.860863 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:30.860838    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25a681-pmq2z" event={"ID":"209627bf-2a35-4b35-91d6-e3d818149f42","Type":"ContainerDied","Data":"9ef9cc9a99c9db6d852c7defe4b5f045656ef25454da335a884ed37c13e53e58"}
Sep 10 05:06:30.861099 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:30.860878    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ef9cc9a99c9db6d852c7defe4b5f045656ef25454da335a884ed37c13e53e58"
Sep 10 05:06:37.681636 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:37.681598    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"]
Sep 10 05:06:37.700638 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:37.700605    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:38.616594 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:38.616558    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"]
Sep 10 05:06:38.636320 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:38.636284    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:42.656554 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:42.656509    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:42.725184 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:42.725153    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h"]
Sep 10 05:06:42.725471 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:42.725410    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h" podUID="f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5" containerName="payload-pre-processing" containerID="cri-o://fff822e085f0e224dc4c7249cb576ad66a20fe0e0d3fadc896c97413c2c30ee9" gracePeriod=30
Sep 10 05:06:42.727673 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:42.727644    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72"]
Sep 10 05:06:42.727951 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:42.727901    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72" podUID="5efb1652-c513-47f7-be66-51c2bb329578" containerName="payload-processing" containerID="cri-o://cc5c2323b94797c596ac48fd675d976a443ef096c6de5d408ffe2bdd554e76e1" gracePeriod=30
Sep 10 05:06:42.774313 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:42.774275    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:42.869197 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:42.869132    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:42.926365 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:42.926338    2575 generic.go:356] "Generic (PLEG): container finished" podID="5efb1652-c513-47f7-be66-51c2bb329578" containerID="cc5c2323b94797c596ac48fd675d976a443ef096c6de5d408ffe2bdd554e76e1" exitCode=0
Sep 10 05:06:42.926506 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:42.926412    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72" event={"ID":"5efb1652-c513-47f7-be66-51c2bb329578","Type":"ContainerDied","Data":"cc5c2323b94797c596ac48fd675d976a443ef096c6de5d408ffe2bdd554e76e1"}
Sep 10 05:06:42.928221 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:42.928197    2575 generic.go:356] "Generic (PLEG): container finished" podID="f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5" containerID="fff822e085f0e224dc4c7249cb576ad66a20fe0e0d3fadc896c97413c2c30ee9" exitCode=0
Sep 10 05:06:42.928351 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:42.928245    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h" event={"ID":"f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5","Type":"ContainerDied","Data":"fff822e085f0e224dc4c7249cb576ad66a20fe0e0d3fadc896c97413c2c30ee9"}
Sep 10 05:06:42.957096 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:42.957039    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25a681-pmq2z"]
Sep 10 05:06:42.961198 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:42.961157    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:42.961878 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:42.961858    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25a681-pmq2z"]
Sep 10 05:06:43.004196 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.004172    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h"
Sep 10 05:06:43.006819 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.006800    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72"
Sep 10 05:06:43.081296 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.081261    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5-plugins-config-volume\") pod \"f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5\" (UID: \"f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5\") "
Sep 10 05:06:43.081474 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.081353    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5efb1652-c513-47f7-be66-51c2bb329578-plugins-config-volume\") pod \"5efb1652-c513-47f7-be66-51c2bb329578\" (UID: \"5efb1652-c513-47f7-be66-51c2bb329578\") "
Sep 10 05:06:43.081474 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.081386    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jfqn\" (UniqueName: \"kubernetes.io/projected/f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5-kube-api-access-9jfqn\") pod \"f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5\" (UID: \"f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5\") "
Sep 10 05:06:43.081474 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.081403    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwb9t\" (UniqueName: \"kubernetes.io/projected/5efb1652-c513-47f7-be66-51c2bb329578-kube-api-access-hwb9t\") pod \"5efb1652-c513-47f7-be66-51c2bb329578\" (UID: \"5efb1652-c513-47f7-be66-51c2bb329578\") "
Sep 10 05:06:43.081758 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.081728    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5" (UID: "f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:06:43.081821 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.081736    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5efb1652-c513-47f7-be66-51c2bb329578-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5efb1652-c513-47f7-be66-51c2bb329578" (UID: "5efb1652-c513-47f7-be66-51c2bb329578"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:06:43.083579 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.083551    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5efb1652-c513-47f7-be66-51c2bb329578-kube-api-access-hwb9t" (OuterVolumeSpecName: "kube-api-access-hwb9t") pod "5efb1652-c513-47f7-be66-51c2bb329578" (UID: "5efb1652-c513-47f7-be66-51c2bb329578"). InnerVolumeSpecName "kube-api-access-hwb9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:06:43.083656 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.083580    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5-kube-api-access-9jfqn" (OuterVolumeSpecName: "kube-api-access-9jfqn") pod "f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5" (UID: "f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5"). InnerVolumeSpecName "kube-api-access-9jfqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:06:43.182856 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.182759    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5efb1652-c513-47f7-be66-51c2bb329578-plugins-config-volume\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:06:43.182856 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.182803    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jfqn\" (UniqueName: \"kubernetes.io/projected/f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5-kube-api-access-9jfqn\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:06:43.182856 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.182813    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwb9t\" (UniqueName: \"kubernetes.io/projected/5efb1652-c513-47f7-be66-51c2bb329578-kube-api-access-hwb9t\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:06:43.182856 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.182824    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5-plugins-config-volume\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:06:43.335169 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.335125    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:43.403980 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.403941    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:43.862783 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.862751    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="209627bf-2a35-4b35-91d6-e3d818149f42" path="/var/lib/kubelet/pods/209627bf-2a35-4b35-91d6-e3d818149f42/volumes"
Sep 10 05:06:43.934494 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.934462    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72"
Sep 10 05:06:43.934695 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.934503    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72" event={"ID":"5efb1652-c513-47f7-be66-51c2bb329578","Type":"ContainerDied","Data":"5915fa142d17ee2747dfd18cbc63afd51d667bd12362d2efcdd03ca34f102aa2"}
Sep 10 05:06:43.934695 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.934570    2575 scope.go:117] "RemoveContainer" containerID="cc5c2323b94797c596ac48fd675d976a443ef096c6de5d408ffe2bdd554e76e1"
Sep 10 05:06:43.936207 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.936187    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h"
Sep 10 05:06:43.936308 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.936203    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h" event={"ID":"f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5","Type":"ContainerDied","Data":"db39d374586a0199c761be778fb75231befc6b2c0640c4417af45463409be873"}
Sep 10 05:06:43.943862 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.943843    2575 scope.go:117] "RemoveContainer" containerID="fff822e085f0e224dc4c7249cb576ad66a20fe0e0d3fadc896c97413c2c30ee9"
Sep 10 05:06:43.953426 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.953403    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h"]
Sep 10 05:06:43.957889 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.957864    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-25a681-58f5db578-5pw7h"]
Sep 10 05:06:43.969077 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.969038    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72"]
Sep 10 05:06:43.971108 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:43.971086    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-25a681-589db85f95-hgg72"]
Sep 10 05:06:45.862512 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:45.862411    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5efb1652-c513-47f7-be66-51c2bb329578" path="/var/lib/kubelet/pods/5efb1652-c513-47f7-be66-51c2bb329578/volumes"
Sep 10 05:06:45.881156 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:45.862745    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5" path="/var/lib/kubelet/pods/f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5/volumes"
Sep 10 05:06:46.406684 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.406650    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"]
Sep 10 05:06:46.406991 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.406980    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="209627bf-2a35-4b35-91d6-e3d818149f42" containerName="revoke-keys"
Sep 10 05:06:46.407036 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.406993    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="209627bf-2a35-4b35-91d6-e3d818149f42" containerName="revoke-keys"
Sep 10 05:06:46.407036 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.407016    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5efb1652-c513-47f7-be66-51c2bb329578" containerName="payload-processing"
Sep 10 05:06:46.407036 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.407021    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5efb1652-c513-47f7-be66-51c2bb329578" containerName="payload-processing"
Sep 10 05:06:46.407036 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.407028    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5" containerName="payload-pre-processing"
Sep 10 05:06:46.407036 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.407034    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5" containerName="payload-pre-processing"
Sep 10 05:06:46.407214 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.407103    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5efb1652-c513-47f7-be66-51c2bb329578" containerName="payload-processing"
Sep 10 05:06:46.407214 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.407113    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="209627bf-2a35-4b35-91d6-e3d818149f42" containerName="revoke-keys"
Sep 10 05:06:46.407214 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.407122    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8cc1fdf-ec26-48fb-9596-24c2a24fc5e5" containerName="payload-pre-processing"
Sep 10 05:06:46.410161 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.410135    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:46.412742 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.412722    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4bfef6\"/\"rate-test-model-4bfef6-kserve-self-signed-certs\""
Sep 10 05:06:46.419602 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.419579    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"]
Sep 10 05:06:46.438894 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.438865    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:46.511867 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.511830    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-tls-certs\") pod \"rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:46.512030 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.511884    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-855vh\" (UniqueName: \"kubernetes.io/projected/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-kube-api-access-855vh\") pod \"rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:46.512030 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.511936    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-dshm\") pod \"rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:46.512030 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.511965    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-model-cache\") pod \"rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:46.512030 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.512001    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-tmp-dir\") pod \"rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:46.512030 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.512026    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-home\") pod \"rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:46.613378 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.613296    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-tls-certs\") pod \"rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:46.613378 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.613340    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-855vh\" (UniqueName: \"kubernetes.io/projected/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-kube-api-access-855vh\") pod \"rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:46.613378 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.613368    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-dshm\") pod \"rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:46.613640 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.613407    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-model-cache\") pod \"rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:46.613640 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.613434    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-tmp-dir\") pod \"rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:46.613640 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.613467    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-home\") pod \"rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:46.613936 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.613891    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-model-cache\") pod \"rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:46.613936 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.613930    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-tmp-dir\") pod \"rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:46.614147 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.613945    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-home\") pod \"rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:46.615812 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.615792    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-dshm\") pod \"rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:46.615993 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.615977    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-tls-certs\") pod \"rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:46.621541 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.621520    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-855vh\" (UniqueName: \"kubernetes.io/projected/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-kube-api-access-855vh\") pod \"rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") " pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:46.720509 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.720471    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:46.990956 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:46.990929    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"]
Sep 10 05:06:46.993422 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:06:46.993395    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad02a8cb_3cfb_40f4_b812_bd7e71777c5b.slice/crio-018ca86bc89e81f11a316fbdd78c27530eb6a89164988b75c5d4e72057203acf WatchSource:0}: Error finding container 018ca86bc89e81f11a316fbdd78c27530eb6a89164988b75c5d4e72057203acf: Status 404 returned error can't find the container with id 018ca86bc89e81f11a316fbdd78c27530eb6a89164988b75c5d4e72057203acf
Sep 10 05:06:47.959598 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:47.959567    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk" event={"ID":"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b","Type":"ContainerStarted","Data":"a82fbede14db8bce410b53b8ea51650141d5aa56f1b9cd3193704b2e31e2fdbd"}
Sep 10 05:06:47.959773 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:47.959610    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk" event={"ID":"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b","Type":"ContainerStarted","Data":"018ca86bc89e81f11a316fbdd78c27530eb6a89164988b75c5d4e72057203acf"}
Sep 10 05:06:47.977386 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:47.977339    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk" podStartSLOduration=1.977320748 podStartE2EDuration="1.977320748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:06:47.975900552 +0000 UTC m=+1544.737669747" watchObservedRunningTime="2026-09-10 05:06:47.977320748 +0000 UTC m=+1544.739089931"
Sep 10 05:06:48.534326 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:48.534283    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:49.854885 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:49.854847    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:49.952082 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:49.952028    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25bd47-gn9vk"]
Sep 10 05:06:49.955450 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:49.955432    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25bd47-gn9vk"
Sep 10 05:06:49.957999 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:49.957977    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dkf6w\""
Sep 10 05:06:49.957999 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:49.957986    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 05:06:49.962849 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:49.962824    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25bd47-gn9vk"]
Sep 10 05:06:50.041808 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:50.041771    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99064bbd-6eb6-485c-8ca7-2d42171a70b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-25bd47-gn9vk\" (UID: \"99064bbd-6eb6-485c-8ca7-2d42171a70b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25bd47-gn9vk"
Sep 10 05:06:50.070400 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:50.070363    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:06:50.143106 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:50.142997    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99064bbd-6eb6-485c-8ca7-2d42171a70b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-25bd47-gn9vk\" (UID: \"99064bbd-6eb6-485c-8ca7-2d42171a70b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25bd47-gn9vk"
Sep 10 05:06:50.143668 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:50.143644    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99064bbd-6eb6-485c-8ca7-2d42171a70b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-25bd47-gn9vk\" (UID: \"99064bbd-6eb6-485c-8ca7-2d42171a70b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25bd47-gn9vk"
Sep 10 05:06:50.267464 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:50.267420    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25bd47-gn9vk"
Sep 10 05:06:50.541041 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:50.541014    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25bd47-gn9vk"]
Sep 10 05:06:50.543144 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:06:50.543107    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99064bbd_6eb6_485c_8ca7_2d42171a70b9.slice/crio-651e6cfb847f82d62d620f7f86f89f3b3509f73f763f33827ce73db82b50692c WatchSource:0}: Error finding container 651e6cfb847f82d62d620f7f86f89f3b3509f73f763f33827ce73db82b50692c: Status 404 returned error can't find the container with id 651e6cfb847f82d62d620f7f86f89f3b3509f73f763f33827ce73db82b50692c
Sep 10 05:06:50.975750 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:50.975708    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25bd47-gn9vk" event={"ID":"99064bbd-6eb6-485c-8ca7-2d42171a70b9","Type":"ContainerStarted","Data":"4e589f1d97342389780024e620f22cdb7272ab70a0699628c6b3417afca1cf53"}
Sep 10 05:06:50.976144 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:50.975780    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25bd47-gn9vk" event={"ID":"99064bbd-6eb6-485c-8ca7-2d42171a70b9","Type":"ContainerStarted","Data":"651e6cfb847f82d62d620f7f86f89f3b3509f73f763f33827ce73db82b50692c"}
Sep 10 05:06:50.991229 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:50.991171    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25bd47-gn9vk" podStartSLOduration=1.991155878 podStartE2EDuration="1.991155878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:06:50.991046347 +0000 UTC m=+1547.752815530" watchObservedRunningTime="2026-09-10 05:06:50.991155878 +0000 UTC m=+1547.752925059"
Sep 10 05:06:51.981591 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:51.981557    2575 generic.go:356] "Generic (PLEG): container finished" podID="99064bbd-6eb6-485c-8ca7-2d42171a70b9" containerID="4e589f1d97342389780024e620f22cdb7272ab70a0699628c6b3417afca1cf53" exitCode=0
Sep 10 05:06:51.981977 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:51.981648    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25bd47-gn9vk" event={"ID":"99064bbd-6eb6-485c-8ca7-2d42171a70b9","Type":"ContainerDied","Data":"4e589f1d97342389780024e620f22cdb7272ab70a0699628c6b3417afca1cf53"}
Sep 10 05:06:52.701012 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:52.700960    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54" podUID="56edcfeb-2bfa-4b0b-b243-e623697ac986" containerName="main" containerID="cri-o://d55f2ac072dd40e9bda7a1ee9b9bd9ab99a0259a840c1c4f1cd74da4b57404e8" gracePeriod=45
Sep 10 05:06:52.948013 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:52.947986    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:06:52.988166 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:52.988061    2575 generic.go:356] "Generic (PLEG): container finished" podID="56edcfeb-2bfa-4b0b-b243-e623697ac986" containerID="d55f2ac072dd40e9bda7a1ee9b9bd9ab99a0259a840c1c4f1cd74da4b57404e8" exitCode=0
Sep 10 05:06:52.988166 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:52.988118    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"
Sep 10 05:06:52.988624 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:52.988169    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54" event={"ID":"56edcfeb-2bfa-4b0b-b243-e623697ac986","Type":"ContainerDied","Data":"d55f2ac072dd40e9bda7a1ee9b9bd9ab99a0259a840c1c4f1cd74da4b57404e8"}
Sep 10 05:06:52.988624 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:52.988223    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54" event={"ID":"56edcfeb-2bfa-4b0b-b243-e623697ac986","Type":"ContainerDied","Data":"0cdf2fbafb5a0af69ceb58023ff6103fbff2974b59f5e9069eecb18352a95f49"}
Sep 10 05:06:52.988624 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:52.988252    2575 scope.go:117] "RemoveContainer" containerID="d55f2ac072dd40e9bda7a1ee9b9bd9ab99a0259a840c1c4f1cd74da4b57404e8"
Sep 10 05:06:52.997332 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:52.997311    2575 scope.go:117] "RemoveContainer" containerID="d55f2ac072dd40e9bda7a1ee9b9bd9ab99a0259a840c1c4f1cd74da4b57404e8"
Sep 10 05:06:52.997712 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:06:52.997684    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d55f2ac072dd40e9bda7a1ee9b9bd9ab99a0259a840c1c4f1cd74da4b57404e8\": container with ID starting with d55f2ac072dd40e9bda7a1ee9b9bd9ab99a0259a840c1c4f1cd74da4b57404e8 not found: ID does not exist" containerID="d55f2ac072dd40e9bda7a1ee9b9bd9ab99a0259a840c1c4f1cd74da4b57404e8"
Sep 10 05:06:52.997870 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:52.997714    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d55f2ac072dd40e9bda7a1ee9b9bd9ab99a0259a840c1c4f1cd74da4b57404e8"} err="failed to get container status \"d55f2ac072dd40e9bda7a1ee9b9bd9ab99a0259a840c1c4f1cd74da4b57404e8\": rpc error: code = NotFound desc = could not find container \"d55f2ac072dd40e9bda7a1ee9b9bd9ab99a0259a840c1c4f1cd74da4b57404e8\": container with ID starting with d55f2ac072dd40e9bda7a1ee9b9bd9ab99a0259a840c1c4f1cd74da4b57404e8 not found: ID does not exist"
Sep 10 05:06:53.072475 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.072442    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-dshm\") pod \"56edcfeb-2bfa-4b0b-b243-e623697ac986\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") "
Sep 10 05:06:53.072653 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.072498    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-model-cache\") pod \"56edcfeb-2bfa-4b0b-b243-e623697ac986\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") "
Sep 10 05:06:53.072653 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.072530    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-home\") pod \"56edcfeb-2bfa-4b0b-b243-e623697ac986\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") "
Sep 10 05:06:53.072653 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.072595    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6zph\" (UniqueName: \"kubernetes.io/projected/56edcfeb-2bfa-4b0b-b243-e623697ac986-kube-api-access-h6zph\") pod \"56edcfeb-2bfa-4b0b-b243-e623697ac986\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") "
Sep 10 05:06:53.072653 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.072639    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-tmp-dir\") pod \"56edcfeb-2bfa-4b0b-b243-e623697ac986\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") "
Sep 10 05:06:53.072882 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.072707    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56edcfeb-2bfa-4b0b-b243-e623697ac986-tls-certs\") pod \"56edcfeb-2bfa-4b0b-b243-e623697ac986\" (UID: \"56edcfeb-2bfa-4b0b-b243-e623697ac986\") "
Sep 10 05:06:53.072992 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.072966    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-home" (OuterVolumeSpecName: "home") pod "56edcfeb-2bfa-4b0b-b243-e623697ac986" (UID: "56edcfeb-2bfa-4b0b-b243-e623697ac986"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:06:53.073043 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.072968    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-model-cache" (OuterVolumeSpecName: "model-cache") pod "56edcfeb-2bfa-4b0b-b243-e623697ac986" (UID: "56edcfeb-2bfa-4b0b-b243-e623697ac986"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:06:53.073043 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.073012    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56edcfeb-2bfa-4b0b-b243-e623697ac986" (UID: "56edcfeb-2bfa-4b0b-b243-e623697ac986"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:06:53.074847 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.074824    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-dshm" (OuterVolumeSpecName: "dshm") pod "56edcfeb-2bfa-4b0b-b243-e623697ac986" (UID: "56edcfeb-2bfa-4b0b-b243-e623697ac986"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:06:53.074960 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.074846    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56edcfeb-2bfa-4b0b-b243-e623697ac986-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56edcfeb-2bfa-4b0b-b243-e623697ac986" (UID: "56edcfeb-2bfa-4b0b-b243-e623697ac986"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 05:06:53.074960 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.074858    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56edcfeb-2bfa-4b0b-b243-e623697ac986-kube-api-access-h6zph" (OuterVolumeSpecName: "kube-api-access-h6zph") pod "56edcfeb-2bfa-4b0b-b243-e623697ac986" (UID: "56edcfeb-2bfa-4b0b-b243-e623697ac986"). InnerVolumeSpecName "kube-api-access-h6zph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:06:53.107659 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.107636    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25bd47-gn9vk"
Sep 10 05:06:53.174171 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.174137    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-model-cache\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:06:53.174171 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.174164    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-home\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:06:53.174171 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.174175    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6zph\" (UniqueName: \"kubernetes.io/projected/56edcfeb-2bfa-4b0b-b243-e623697ac986-kube-api-access-h6zph\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:06:53.174401 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.174184    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-tmp-dir\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:06:53.174401 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.174193    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56edcfeb-2bfa-4b0b-b243-e623697ac986-tls-certs\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:06:53.174401 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.174201    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56edcfeb-2bfa-4b0b-b243-e623697ac986-dshm\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:06:53.275123 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.275042    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99064bbd-6eb6-485c-8ca7-2d42171a70b9-maas-api-service-ca\") pod \"99064bbd-6eb6-485c-8ca7-2d42171a70b9\" (UID: \"99064bbd-6eb6-485c-8ca7-2d42171a70b9\") "
Sep 10 05:06:53.275413 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.275389    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99064bbd-6eb6-485c-8ca7-2d42171a70b9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "99064bbd-6eb6-485c-8ca7-2d42171a70b9" (UID: "99064bbd-6eb6-485c-8ca7-2d42171a70b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:06:53.309033 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.309000    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"]
Sep 10 05:06:53.312795 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.312774    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4bfef6/sub-test-model-4bfef6-kserve-86cb5d7dfc-26p54"]
Sep 10 05:06:53.376325 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.376293    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99064bbd-6eb6-485c-8ca7-2d42171a70b9-maas-api-service-ca\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:06:53.636098 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.635987    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh" podUID="820e5f0d-7fcb-4ef4-adc6-0d62fa16491c" containerName="main" containerID="cri-o://81baf6c04a5207208e255b55f3f3aee42835fb9b75d95a58749ef98076e40384" gracePeriod=45
Sep 10 05:06:53.865108 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.865048    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56edcfeb-2bfa-4b0b-b243-e623697ac986" path="/var/lib/kubelet/pods/56edcfeb-2bfa-4b0b-b243-e623697ac986/volumes"
Sep 10 05:06:53.892045 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.892022    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:06:53.995527 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.995495    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25bd47-gn9vk"
Sep 10 05:06:53.995946 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.995530    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25bd47-gn9vk" event={"ID":"99064bbd-6eb6-485c-8ca7-2d42171a70b9","Type":"ContainerDied","Data":"651e6cfb847f82d62d620f7f86f89f3b3509f73f763f33827ce73db82b50692c"}
Sep 10 05:06:53.995946 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.995577    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="651e6cfb847f82d62d620f7f86f89f3b3509f73f763f33827ce73db82b50692c"
Sep 10 05:06:53.997854 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.997831    2575 generic.go:356] "Generic (PLEG): container finished" podID="820e5f0d-7fcb-4ef4-adc6-0d62fa16491c" containerID="81baf6c04a5207208e255b55f3f3aee42835fb9b75d95a58749ef98076e40384" exitCode=0
Sep 10 05:06:53.997970 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.997861    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh" event={"ID":"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c","Type":"ContainerDied","Data":"81baf6c04a5207208e255b55f3f3aee42835fb9b75d95a58749ef98076e40384"}
Sep 10 05:06:53.997970 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.997881    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"
Sep 10 05:06:53.997970 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.997893    2575 scope.go:117] "RemoveContainer" containerID="81baf6c04a5207208e255b55f3f3aee42835fb9b75d95a58749ef98076e40384"
Sep 10 05:06:53.998162 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:53.997883    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh" event={"ID":"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c","Type":"ContainerDied","Data":"594a8c4efcfda6b32c1cd9ab0206bdb9d7754cbdaf664c3370edb6e427741632"}
Sep 10 05:06:54.006139 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:54.006049    2575 scope.go:117] "RemoveContainer" containerID="81baf6c04a5207208e255b55f3f3aee42835fb9b75d95a58749ef98076e40384"
Sep 10 05:06:54.006375 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:06:54.006357    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81baf6c04a5207208e255b55f3f3aee42835fb9b75d95a58749ef98076e40384\": container with ID starting with 81baf6c04a5207208e255b55f3f3aee42835fb9b75d95a58749ef98076e40384 not found: ID does not exist" containerID="81baf6c04a5207208e255b55f3f3aee42835fb9b75d95a58749ef98076e40384"
Sep 10 05:06:54.006423 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:54.006386    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81baf6c04a5207208e255b55f3f3aee42835fb9b75d95a58749ef98076e40384"} err="failed to get container status \"81baf6c04a5207208e255b55f3f3aee42835fb9b75d95a58749ef98076e40384\": rpc error: code = NotFound desc = could not find container \"81baf6c04a5207208e255b55f3f3aee42835fb9b75d95a58749ef98076e40384\": container with ID starting with 81baf6c04a5207208e255b55f3f3aee42835fb9b75d95a58749ef98076e40384 not found: ID does not exist"
Sep 10 05:06:54.082100 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:54.082046    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-tmp-dir\") pod \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") "
Sep 10 05:06:54.082235 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:54.082109    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-model-cache\") pod \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") "
Sep 10 05:06:54.082235 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:54.082163    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-tls-certs\") pod \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") "
Sep 10 05:06:54.082235 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:54.082193    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpdrv\" (UniqueName: \"kubernetes.io/projected/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-kube-api-access-tpdrv\") pod \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") "
Sep 10 05:06:54.082381 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:54.082238    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-home\") pod \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") "
Sep 10 05:06:54.082381 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:54.082257    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-dshm\") pod \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\" (UID: \"820e5f0d-7fcb-4ef4-adc6-0d62fa16491c\") "
Sep 10 05:06:54.082381 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:54.082344    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "820e5f0d-7fcb-4ef4-adc6-0d62fa16491c" (UID: "820e5f0d-7fcb-4ef4-adc6-0d62fa16491c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:06:54.082502 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:54.082398    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-model-cache" (OuterVolumeSpecName: "model-cache") pod "820e5f0d-7fcb-4ef4-adc6-0d62fa16491c" (UID: "820e5f0d-7fcb-4ef4-adc6-0d62fa16491c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:06:54.082561 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:54.082508    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-home" (OuterVolumeSpecName: "home") pod "820e5f0d-7fcb-4ef4-adc6-0d62fa16491c" (UID: "820e5f0d-7fcb-4ef4-adc6-0d62fa16491c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:06:54.082561 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:54.082513    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-model-cache\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:06:54.082561 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:54.082540    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-tmp-dir\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:06:54.084371 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:54.084338    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-kube-api-access-tpdrv" (OuterVolumeSpecName: "kube-api-access-tpdrv") pod "820e5f0d-7fcb-4ef4-adc6-0d62fa16491c" (UID: "820e5f0d-7fcb-4ef4-adc6-0d62fa16491c"). InnerVolumeSpecName "kube-api-access-tpdrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:06:54.084470 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:54.084377    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "820e5f0d-7fcb-4ef4-adc6-0d62fa16491c" (UID: "820e5f0d-7fcb-4ef4-adc6-0d62fa16491c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 05:06:54.084470 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:54.084376    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-dshm" (OuterVolumeSpecName: "dshm") pod "820e5f0d-7fcb-4ef4-adc6-0d62fa16491c" (UID: "820e5f0d-7fcb-4ef4-adc6-0d62fa16491c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:06:54.183266 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:54.183188    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-home\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:06:54.183266 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:54.183214    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-dshm\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:06:54.183266 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:54.183223    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-tls-certs\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:06:54.183266 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:54.183233    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpdrv\" (UniqueName: \"kubernetes.io/projected/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c-kube-api-access-tpdrv\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:06:54.318946 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:54.318916    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"]
Sep 10 05:06:54.322487 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:54.322465    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b702d/sub-test-model-3b702d-kserve-57b84d486c-qblbh"]
Sep 10 05:06:55.862620 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:55.862585    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="820e5f0d-7fcb-4ef4-adc6-0d62fa16491c" path="/var/lib/kubelet/pods/820e5f0d-7fcb-4ef4-adc6-0d62fa16491c/volumes"
Sep 10 05:06:56.720868 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:56.720831    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:56.721097 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:56.720918    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:56.733719 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:56.733692    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:06:57.025227 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:06:57.025139    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:07:00.601354 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:00.601317    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:00.714841 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:00.714804    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:00.849439 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:00.849405    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:00.907738 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:00.907648    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25bd47-gn9vk"]
Sep 10 05:07:00.910224 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:00.910199    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25bd47-gn9vk"]
Sep 10 05:07:00.919421 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:00.919395    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:01.282626 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:01.282550    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:01.862011 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:01.861977    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99064bbd-6eb6-485c-8ca7-2d42171a70b9" path="/var/lib/kubelet/pods/99064bbd-6eb6-485c-8ca7-2d42171a70b9/volumes"
Sep 10 05:07:03.278597 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:03.278565    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:04.122535 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:04.122500    2575 scope.go:117] "RemoveContainer" containerID="16e28ade335d04948d5dccfc16e88acec65227b0a42b789d5114f0e140cb3fab"
Sep 10 05:07:04.134264 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:04.134238    2575 scope.go:117] "RemoveContainer" containerID="668550ae2bf7f48d60e24422e1ddbbe1178ef6509d8e61fd2fb135c591b2452a"
Sep 10 05:07:07.094183 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:07.094147    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:07.195748 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:07.195714    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:07.337128 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:07.337085    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:07.678388 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:07.678350    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97ffde-p77qt"]
Sep 10 05:07:07.678733 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:07.678721    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56edcfeb-2bfa-4b0b-b243-e623697ac986" containerName="main"
Sep 10 05:07:07.678775 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:07.678735    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="56edcfeb-2bfa-4b0b-b243-e623697ac986" containerName="main"
Sep 10 05:07:07.678775 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:07.678751    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99064bbd-6eb6-485c-8ca7-2d42171a70b9" containerName="revoke-keys"
Sep 10 05:07:07.678775 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:07.678757    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="99064bbd-6eb6-485c-8ca7-2d42171a70b9" containerName="revoke-keys"
Sep 10 05:07:07.678775 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:07.678774    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="820e5f0d-7fcb-4ef4-adc6-0d62fa16491c" containerName="main"
Sep 10 05:07:07.678905 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:07.678780    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="820e5f0d-7fcb-4ef4-adc6-0d62fa16491c" containerName="main"
Sep 10 05:07:07.678905 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:07.678830    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="820e5f0d-7fcb-4ef4-adc6-0d62fa16491c" containerName="main"
Sep 10 05:07:07.678905 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:07.678838    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="99064bbd-6eb6-485c-8ca7-2d42171a70b9" containerName="revoke-keys"
Sep 10 05:07:07.678905 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:07.678845    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="56edcfeb-2bfa-4b0b-b243-e623697ac986" containerName="main"
Sep 10 05:07:07.681696 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:07.681679    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97ffde-p77qt"
Sep 10 05:07:07.685038 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:07.685015    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 05:07:07.685105 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:07.685081    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dkf6w\""
Sep 10 05:07:07.690874 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:07.690853    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97ffde-p77qt"]
Sep 10 05:07:07.696818 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:07.696791    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a77bb99d-d677-4b43-a4d1-b37c95d00a4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-97ffde-p77qt\" (UID: \"a77bb99d-d677-4b43-a4d1-b37c95d00a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97ffde-p77qt"
Sep 10 05:07:07.797608 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:07.797571    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a77bb99d-d677-4b43-a4d1-b37c95d00a4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-97ffde-p77qt\" (UID: \"a77bb99d-d677-4b43-a4d1-b37c95d00a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97ffde-p77qt"
Sep 10 05:07:07.798222 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:07.798204    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a77bb99d-d677-4b43-a4d1-b37c95d00a4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-97ffde-p77qt\" (UID: \"a77bb99d-d677-4b43-a4d1-b37c95d00a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97ffde-p77qt"
Sep 10 05:07:07.917120 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:07.917054    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:07.993155 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:07.993120    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97ffde-p77qt"
Sep 10 05:07:08.266783 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:08.266699    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97ffde-p77qt"]
Sep 10 05:07:08.270454 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:07:08.270424    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77bb99d_d677_4b43_a4d1_b37c95d00a4e.slice/crio-a69cc37607d3463a8d759a9d7475f36bd0eff23aaadbc29839fec16f7d411366 WatchSource:0}: Error finding container a69cc37607d3463a8d759a9d7475f36bd0eff23aaadbc29839fec16f7d411366: Status 404 returned error can't find the container with id a69cc37607d3463a8d759a9d7475f36bd0eff23aaadbc29839fec16f7d411366
Sep 10 05:07:09.090333 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:09.090299    2575 generic.go:356] "Generic (PLEG): container finished" podID="a77bb99d-d677-4b43-a4d1-b37c95d00a4e" containerID="0890b387103175094b576d4e43c0a60783fabf0e559a4ec4f760fb6388bdb589" exitCode=0
Sep 10 05:07:09.090531 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:09.090369    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97ffde-p77qt" event={"ID":"a77bb99d-d677-4b43-a4d1-b37c95d00a4e","Type":"ContainerDied","Data":"0890b387103175094b576d4e43c0a60783fabf0e559a4ec4f760fb6388bdb589"}
Sep 10 05:07:09.090531 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:09.090415    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97ffde-p77qt" event={"ID":"a77bb99d-d677-4b43-a4d1-b37c95d00a4e","Type":"ContainerStarted","Data":"a69cc37607d3463a8d759a9d7475f36bd0eff23aaadbc29839fec16f7d411366"}
Sep 10 05:07:10.027356 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:10.027305    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:10.221800 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:10.221772    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97ffde-p77qt"
Sep 10 05:07:10.321636 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:10.321544    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a77bb99d-d677-4b43-a4d1-b37c95d00a4e-maas-api-service-ca\") pod \"a77bb99d-d677-4b43-a4d1-b37c95d00a4e\" (UID: \"a77bb99d-d677-4b43-a4d1-b37c95d00a4e\") "
Sep 10 05:07:10.321951 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:10.321922    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a77bb99d-d677-4b43-a4d1-b37c95d00a4e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a77bb99d-d677-4b43-a4d1-b37c95d00a4e" (UID: "a77bb99d-d677-4b43-a4d1-b37c95d00a4e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:07:10.422964 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:10.422928    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a77bb99d-d677-4b43-a4d1-b37c95d00a4e-maas-api-service-ca\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:07:11.102711 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:11.102681    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97ffde-p77qt"
Sep 10 05:07:11.103129 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:11.102711    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97ffde-p77qt" event={"ID":"a77bb99d-d677-4b43-a4d1-b37c95d00a4e","Type":"ContainerDied","Data":"a69cc37607d3463a8d759a9d7475f36bd0eff23aaadbc29839fec16f7d411366"}
Sep 10 05:07:11.103129 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:11.102751    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a69cc37607d3463a8d759a9d7475f36bd0eff23aaadbc29839fec16f7d411366"
Sep 10 05:07:15.601530 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:15.601492    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:17.819799 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:17.819762    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:18.443307 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:18.443271    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx"]
Sep 10 05:07:18.443514 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:18.443487    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx" podUID="291d939f-8e6b-49cd-bc39-478c9f952983" containerName="payload-processing" containerID="cri-o://fe85cf863ac84e920aabeb1483cd051365b5fa61bbe5ce750c4cff87da5973a9" gracePeriod=30
Sep 10 05:07:18.610039 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:18.610007    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97ffde-p77qt"]
Sep 10 05:07:18.612299 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:18.612274    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97ffde-p77qt"]
Sep 10 05:07:18.649499 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:18.649457    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:18.693480 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:18.693427    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx"
Sep 10 05:07:18.729037 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:18.729003    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:18.797715 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:18.797682    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d96zs\" (UniqueName: \"kubernetes.io/projected/291d939f-8e6b-49cd-bc39-478c9f952983-kube-api-access-d96zs\") pod \"291d939f-8e6b-49cd-bc39-478c9f952983\" (UID: \"291d939f-8e6b-49cd-bc39-478c9f952983\") "
Sep 10 05:07:18.797715 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:18.797720    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291d939f-8e6b-49cd-bc39-478c9f952983-plugins-config-volume\") pod \"291d939f-8e6b-49cd-bc39-478c9f952983\" (UID: \"291d939f-8e6b-49cd-bc39-478c9f952983\") "
Sep 10 05:07:18.798134 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:18.798108    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/291d939f-8e6b-49cd-bc39-478c9f952983-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "291d939f-8e6b-49cd-bc39-478c9f952983" (UID: "291d939f-8e6b-49cd-bc39-478c9f952983"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:07:18.799824 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:18.799801    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/291d939f-8e6b-49cd-bc39-478c9f952983-kube-api-access-d96zs" (OuterVolumeSpecName: "kube-api-access-d96zs") pod "291d939f-8e6b-49cd-bc39-478c9f952983" (UID: "291d939f-8e6b-49cd-bc39-478c9f952983"). InnerVolumeSpecName "kube-api-access-d96zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:07:18.898745 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:18.898706    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d96zs\" (UniqueName: \"kubernetes.io/projected/291d939f-8e6b-49cd-bc39-478c9f952983-kube-api-access-d96zs\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:07:18.898745 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:18.898738    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291d939f-8e6b-49cd-bc39-478c9f952983-plugins-config-volume\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:07:19.006881 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:19.006848    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:19.145560 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:19.145526    2575 generic.go:356] "Generic (PLEG): container finished" podID="291d939f-8e6b-49cd-bc39-478c9f952983" containerID="fe85cf863ac84e920aabeb1483cd051365b5fa61bbe5ce750c4cff87da5973a9" exitCode=0
Sep 10 05:07:19.145748 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:19.145573    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx"
Sep 10 05:07:19.145748 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:19.145610    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx" event={"ID":"291d939f-8e6b-49cd-bc39-478c9f952983","Type":"ContainerDied","Data":"fe85cf863ac84e920aabeb1483cd051365b5fa61bbe5ce750c4cff87da5973a9"}
Sep 10 05:07:19.145748 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:19.145670    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx" event={"ID":"291d939f-8e6b-49cd-bc39-478c9f952983","Type":"ContainerDied","Data":"fd4e707ce3f7f77aed75234eb730f4f1da34a47d35afb962d1b5ff3e658e3e4d"}
Sep 10 05:07:19.145748 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:19.145718    2575 scope.go:117] "RemoveContainer" containerID="fe85cf863ac84e920aabeb1483cd051365b5fa61bbe5ce750c4cff87da5973a9"
Sep 10 05:07:19.154249 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:19.154236    2575 scope.go:117] "RemoveContainer" containerID="fe85cf863ac84e920aabeb1483cd051365b5fa61bbe5ce750c4cff87da5973a9"
Sep 10 05:07:19.154508 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:07:19.154490    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe85cf863ac84e920aabeb1483cd051365b5fa61bbe5ce750c4cff87da5973a9\": container with ID starting with fe85cf863ac84e920aabeb1483cd051365b5fa61bbe5ce750c4cff87da5973a9 not found: ID does not exist" containerID="fe85cf863ac84e920aabeb1483cd051365b5fa61bbe5ce750c4cff87da5973a9"
Sep 10 05:07:19.154555 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:19.154519    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe85cf863ac84e920aabeb1483cd051365b5fa61bbe5ce750c4cff87da5973a9"} err="failed to get container status \"fe85cf863ac84e920aabeb1483cd051365b5fa61bbe5ce750c4cff87da5973a9\": rpc error: code = NotFound desc = could not find container \"fe85cf863ac84e920aabeb1483cd051365b5fa61bbe5ce750c4cff87da5973a9\": container with ID starting with fe85cf863ac84e920aabeb1483cd051365b5fa61bbe5ce750c4cff87da5973a9 not found: ID does not exist"
Sep 10 05:07:19.166372 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:19.166347    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx"]
Sep 10 05:07:19.173832 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:19.173806    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-97ffde-9b9cc7669-kh2fx"]
Sep 10 05:07:19.862719 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:19.862687    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="291d939f-8e6b-49cd-bc39-478c9f952983" path="/var/lib/kubelet/pods/291d939f-8e6b-49cd-bc39-478c9f952983/volumes"
Sep 10 05:07:19.863014 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:19.863001    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a77bb99d-d677-4b43-a4d1-b37c95d00a4e" path="/var/lib/kubelet/pods/a77bb99d-d677-4b43-a4d1-b37c95d00a4e/volumes"
Sep 10 05:07:24.765610 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:24.765564    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:24.834206 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:24.834171    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:25.033056 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:25.032974    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:53.493166 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:53.493132    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"]
Sep 10 05:07:53.514253 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:53.514215    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:54.507869 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:54.507833    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:54.938377 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:54.938288    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:55.251338 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:55.251305    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:55.607207 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:55.607123    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:58.749040 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:58.749002    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:58.848081 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:58.848033    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:59.430095 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:59.430052    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:59.497034 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:59.497001    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:59.730894 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:59.730857    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:07:59.935499 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:07:59.935460    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:08:00.037891 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:00.037800    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:08:00.066982 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:00.066949    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f"]
Sep 10 05:08:00.067330 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:00.067315    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a77bb99d-d677-4b43-a4d1-b37c95d00a4e" containerName="revoke-keys"
Sep 10 05:08:00.067380 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:00.067332    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a77bb99d-d677-4b43-a4d1-b37c95d00a4e" containerName="revoke-keys"
Sep 10 05:08:00.067380 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:00.067349    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="291d939f-8e6b-49cd-bc39-478c9f952983" containerName="payload-processing"
Sep 10 05:08:00.067380 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:00.067356    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="291d939f-8e6b-49cd-bc39-478c9f952983" containerName="payload-processing"
Sep 10 05:08:00.067533 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:00.067422    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="291d939f-8e6b-49cd-bc39-478c9f952983" containerName="payload-processing"
Sep 10 05:08:00.067533 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:00.067433    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a77bb99d-d677-4b43-a4d1-b37c95d00a4e" containerName="revoke-keys"
Sep 10 05:08:00.070512 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:00.070494    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f"
Sep 10 05:08:00.073017 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:00.072994    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-95427a-dockercfg-wxtnl\""
Sep 10 05:08:00.073017 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:00.073008    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-95427a\""
Sep 10 05:08:00.080546 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:00.080523    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f"]
Sep 10 05:08:00.153865 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:00.153836    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbv2s\" (UniqueName: \"kubernetes.io/projected/83a2d361-102b-464b-8d35-8f3df4f969c8-kube-api-access-wbv2s\") pod \"payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f\" (UID: \"83a2d361-102b-464b-8d35-8f3df4f969c8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f"
Sep 10 05:08:00.154033 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:00.153901    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a2d361-102b-464b-8d35-8f3df4f969c8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f\" (UID: \"83a2d361-102b-464b-8d35-8f3df4f969c8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f"
Sep 10 05:08:00.254810 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:00.254776    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbv2s\" (UniqueName: \"kubernetes.io/projected/83a2d361-102b-464b-8d35-8f3df4f969c8-kube-api-access-wbv2s\") pod \"payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f\" (UID: \"83a2d361-102b-464b-8d35-8f3df4f969c8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f"
Sep 10 05:08:00.254983 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:00.254838    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a2d361-102b-464b-8d35-8f3df4f969c8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f\" (UID: \"83a2d361-102b-464b-8d35-8f3df4f969c8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f"
Sep 10 05:08:00.255529 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:00.255510    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a2d361-102b-464b-8d35-8f3df4f969c8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f\" (UID: \"83a2d361-102b-464b-8d35-8f3df4f969c8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f"
Sep 10 05:08:00.263494 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:00.263466    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbv2s\" (UniqueName: \"kubernetes.io/projected/83a2d361-102b-464b-8d35-8f3df4f969c8-kube-api-access-wbv2s\") pod \"payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f\" (UID: \"83a2d361-102b-464b-8d35-8f3df4f969c8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f"
Sep 10 05:08:00.380980 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:00.380893    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f"
Sep 10 05:08:00.650848 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:00.650824    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f"]
Sep 10 05:08:00.653061 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:08:00.653030    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a2d361_102b_464b_8d35_8f3df4f969c8.slice/crio-5c8c6abc5f948f1869a98c5502af5ab389e4f27a72e94438197ea73d83a1ace0 WatchSource:0}: Error finding container 5c8c6abc5f948f1869a98c5502af5ab389e4f27a72e94438197ea73d83a1ace0: Status 404 returned error can't find the container with id 5c8c6abc5f948f1869a98c5502af5ab389e4f27a72e94438197ea73d83a1ace0
Sep 10 05:08:01.343434 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:01.343395    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f" event={"ID":"83a2d361-102b-464b-8d35-8f3df4f969c8","Type":"ContainerStarted","Data":"0d9c1a9ffe30517f6be7692cbe28efe37dfe3b0f9f25576ecb54cfbad980ee19"}
Sep 10 05:08:01.343916 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:01.343473    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f" event={"ID":"83a2d361-102b-464b-8d35-8f3df4f969c8","Type":"ContainerStarted","Data":"5c8c6abc5f948f1869a98c5502af5ab389e4f27a72e94438197ea73d83a1ace0"}
Sep 10 05:08:01.343916 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:01.343502    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f"
Sep 10 05:08:01.361193 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:01.361139    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f" podStartSLOduration=1.361122813 podStartE2EDuration="1.361122813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:08:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:08:01.359360679 +0000 UTC m=+1618.121129863" watchObservedRunningTime="2026-09-10 05:08:01.361122813 +0000 UTC m=+1618.122891994"
Sep 10 05:08:04.170539 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:04.170501    2575 scope.go:117] "RemoveContainer" containerID="ef323bdce5e960f1f2249ac68f1422fdd3fc06be9cce4357dbf60ac2ff0437af"
Sep 10 05:08:04.178897 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:04.178878    2575 scope.go:117] "RemoveContainer" containerID="88acdb212f8584adc358690355fab27678d29c97e8667c3889f511f8dc3f02d3"
Sep 10 05:08:08.512285 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:08.512232    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk" podUID="ad02a8cb-3cfb-40f4-b812-bd7e71777c5b" containerName="main" containerID="cri-o://a82fbede14db8bce410b53b8ea51650141d5aa56f1b9cd3193704b2e31e2fdbd" gracePeriod=45
Sep 10 05:08:08.766420 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:08.766359    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:08:08.835913 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:08.835882    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-dshm\") pod \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") "
Sep 10 05:08:08.836094 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:08.835936    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-855vh\" (UniqueName: \"kubernetes.io/projected/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-kube-api-access-855vh\") pod \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") "
Sep 10 05:08:08.836094 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:08.835966    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-tmp-dir\") pod \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") "
Sep 10 05:08:08.836094 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:08.835997    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-tls-certs\") pod \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") "
Sep 10 05:08:08.836094 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:08.836023    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-home\") pod \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") "
Sep 10 05:08:08.836325 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:08.836174    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-model-cache\") pod \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\" (UID: \"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b\") "
Sep 10 05:08:08.836325 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:08.836298    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-home" (OuterVolumeSpecName: "home") pod "ad02a8cb-3cfb-40f4-b812-bd7e71777c5b" (UID: "ad02a8cb-3cfb-40f4-b812-bd7e71777c5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:08:08.836325 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:08.836311    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad02a8cb-3cfb-40f4-b812-bd7e71777c5b" (UID: "ad02a8cb-3cfb-40f4-b812-bd7e71777c5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:08:08.836451 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:08.836420    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad02a8cb-3cfb-40f4-b812-bd7e71777c5b" (UID: "ad02a8cb-3cfb-40f4-b812-bd7e71777c5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:08:08.836572 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:08.836556    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-tmp-dir\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:08:08.836612 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:08.836578    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-home\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:08:08.836612 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:08.836591    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-model-cache\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:08:08.838192 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:08.838157    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad02a8cb-3cfb-40f4-b812-bd7e71777c5b" (UID: "ad02a8cb-3cfb-40f4-b812-bd7e71777c5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 05:08:08.838617 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:08.838600    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-dshm" (OuterVolumeSpecName: "dshm") pod "ad02a8cb-3cfb-40f4-b812-bd7e71777c5b" (UID: "ad02a8cb-3cfb-40f4-b812-bd7e71777c5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:08:08.838713 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:08.838638    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-kube-api-access-855vh" (OuterVolumeSpecName: "kube-api-access-855vh") pod "ad02a8cb-3cfb-40f4-b812-bd7e71777c5b" (UID: "ad02a8cb-3cfb-40f4-b812-bd7e71777c5b"). InnerVolumeSpecName "kube-api-access-855vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:08:08.937738 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:08.937699    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-855vh\" (UniqueName: \"kubernetes.io/projected/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-kube-api-access-855vh\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:08:08.937738 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:08.937735    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-tls-certs\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:08:08.937941 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:08.937750    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b-dshm\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:08:09.383169 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:09.383132    2575 generic.go:356] "Generic (PLEG): container finished" podID="ad02a8cb-3cfb-40f4-b812-bd7e71777c5b" containerID="a82fbede14db8bce410b53b8ea51650141d5aa56f1b9cd3193704b2e31e2fdbd" exitCode=0
Sep 10 05:08:09.383332 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:09.383175    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"
Sep 10 05:08:09.383332 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:09.383182    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk" event={"ID":"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b","Type":"ContainerDied","Data":"a82fbede14db8bce410b53b8ea51650141d5aa56f1b9cd3193704b2e31e2fdbd"}
Sep 10 05:08:09.383332 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:09.383219    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk" event={"ID":"ad02a8cb-3cfb-40f4-b812-bd7e71777c5b","Type":"ContainerDied","Data":"018ca86bc89e81f11a316fbdd78c27530eb6a89164988b75c5d4e72057203acf"}
Sep 10 05:08:09.383332 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:09.383236    2575 scope.go:117] "RemoveContainer" containerID="a82fbede14db8bce410b53b8ea51650141d5aa56f1b9cd3193704b2e31e2fdbd"
Sep 10 05:08:09.392038 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:09.392020    2575 scope.go:117] "RemoveContainer" containerID="a82fbede14db8bce410b53b8ea51650141d5aa56f1b9cd3193704b2e31e2fdbd"
Sep 10 05:08:09.392344 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:08:09.392324    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a82fbede14db8bce410b53b8ea51650141d5aa56f1b9cd3193704b2e31e2fdbd\": container with ID starting with a82fbede14db8bce410b53b8ea51650141d5aa56f1b9cd3193704b2e31e2fdbd not found: ID does not exist" containerID="a82fbede14db8bce410b53b8ea51650141d5aa56f1b9cd3193704b2e31e2fdbd"
Sep 10 05:08:09.392416 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:09.392354    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82fbede14db8bce410b53b8ea51650141d5aa56f1b9cd3193704b2e31e2fdbd"} err="failed to get container status \"a82fbede14db8bce410b53b8ea51650141d5aa56f1b9cd3193704b2e31e2fdbd\": rpc error: code = NotFound desc = could not find container \"a82fbede14db8bce410b53b8ea51650141d5aa56f1b9cd3193704b2e31e2fdbd\": container with ID starting with a82fbede14db8bce410b53b8ea51650141d5aa56f1b9cd3193704b2e31e2fdbd not found: ID does not exist"
Sep 10 05:08:09.420214 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:09.420187    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"]
Sep 10 05:08:09.430438 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:09.430415    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4bfef6/rate-test-model-4bfef6-kserve-5f8b497dc-d4qmk"]
Sep 10 05:08:09.862422 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:09.862387    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad02a8cb-3cfb-40f4-b812-bd7e71777c5b" path="/var/lib/kubelet/pods/ad02a8cb-3cfb-40f4-b812-bd7e71777c5b/volumes"
Sep 10 05:08:12.352648 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:12.352620    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f"
Sep 10 05:08:23.289844 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:23.289808    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:08:24.119705 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:24.119670    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:08:24.493576 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:24.493544    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:08:24.706430 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:24.706397    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:08:27.494403 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:27.494373    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:08:27.591732 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:27.591684    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:08:28.640715 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:28.640672    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:08:28.741913 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:28.741877    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:08:29.140502 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:29.140468    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:08:29.245582 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:29.245541    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:08:52.503381 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:08:52.503345    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:09:04.206755 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:04.206717    2575 scope.go:117] "RemoveContainer" containerID="e9dff8fd0dc784a6e7ee9f4b2ab159c369177104431897107da91bb07f3dd67e"
Sep 10 05:09:05.596055 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.596022    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"]
Sep 10 05:09:05.596555 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.596401    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad02a8cb-3cfb-40f4-b812-bd7e71777c5b" containerName="main"
Sep 10 05:09:05.596555 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.596414    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad02a8cb-3cfb-40f4-b812-bd7e71777c5b" containerName="main"
Sep 10 05:09:05.596555 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.596489    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad02a8cb-3cfb-40f4-b812-bd7e71777c5b" containerName="main"
Sep 10 05:09:05.599913 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.599884    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:05.603324 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.602851    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-95427a\"/\"default-dockercfg-mtpxj\""
Sep 10 05:09:05.603942 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.603916    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-95427a\"/\"kube-root-ca.crt\""
Sep 10 05:09:05.604107 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.604012    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-95427a\"/\"openshift-service-ca.crt\""
Sep 10 05:09:05.604228 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.604084    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-95427a\"/\"ipp-route-95427a-kserve-self-signed-certs\""
Sep 10 05:09:05.611857 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.611832    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"]
Sep 10 05:09:05.621388 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.621359    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:09:05.728547 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.728512    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-dshm\") pod \"ipp-route-95427a-kserve-77446c58dc-kpvwh\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") " pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:05.728735 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.728554    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-model-cache\") pod \"ipp-route-95427a-kserve-77446c58dc-kpvwh\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") " pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:05.728735 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.728574    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-tmp-dir\") pod \"ipp-route-95427a-kserve-77446c58dc-kpvwh\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") " pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:05.728735 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.728668    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sng2m\" (UniqueName: \"kubernetes.io/projected/9b9764c5-2d00-4242-8121-1401d22e2fe9-kube-api-access-sng2m\") pod \"ipp-route-95427a-kserve-77446c58dc-kpvwh\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") " pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:05.728735 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.728725    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-home\") pod \"ipp-route-95427a-kserve-77446c58dc-kpvwh\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") " pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:05.728941 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.728835    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b9764c5-2d00-4242-8121-1401d22e2fe9-tls-certs\") pod \"ipp-route-95427a-kserve-77446c58dc-kpvwh\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") " pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:05.829384 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.829346    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-home\") pod \"ipp-route-95427a-kserve-77446c58dc-kpvwh\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") " pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:05.829570 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.829450    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b9764c5-2d00-4242-8121-1401d22e2fe9-tls-certs\") pod \"ipp-route-95427a-kserve-77446c58dc-kpvwh\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") " pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:05.829570 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.829480    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-dshm\") pod \"ipp-route-95427a-kserve-77446c58dc-kpvwh\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") " pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:05.829570 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.829511    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-model-cache\") pod \"ipp-route-95427a-kserve-77446c58dc-kpvwh\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") " pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:05.829570 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.829536    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-tmp-dir\") pod \"ipp-route-95427a-kserve-77446c58dc-kpvwh\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") " pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:05.829759 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.829569    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sng2m\" (UniqueName: \"kubernetes.io/projected/9b9764c5-2d00-4242-8121-1401d22e2fe9-kube-api-access-sng2m\") pod \"ipp-route-95427a-kserve-77446c58dc-kpvwh\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") " pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:05.829824 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.829796    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-home\") pod \"ipp-route-95427a-kserve-77446c58dc-kpvwh\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") " pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:05.829900 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.829862    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-model-cache\") pod \"ipp-route-95427a-kserve-77446c58dc-kpvwh\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") " pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:05.829900 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.829890    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-tmp-dir\") pod \"ipp-route-95427a-kserve-77446c58dc-kpvwh\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") " pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:05.831777 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.831752    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-dshm\") pod \"ipp-route-95427a-kserve-77446c58dc-kpvwh\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") " pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:05.832011 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.831996    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b9764c5-2d00-4242-8121-1401d22e2fe9-tls-certs\") pod \"ipp-route-95427a-kserve-77446c58dc-kpvwh\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") " pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:05.837662 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.837640    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sng2m\" (UniqueName: \"kubernetes.io/projected/9b9764c5-2d00-4242-8121-1401d22e2fe9-kube-api-access-sng2m\") pod \"ipp-route-95427a-kserve-77446c58dc-kpvwh\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") " pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:05.910696 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:05.910592    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:06.180042 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:06.180015    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"]
Sep 10 05:09:06.182132 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:09:06.182098    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b9764c5_2d00_4242_8121_1401d22e2fe9.slice/crio-8544688a7d53af049f463e496d85438a3eb85e844ecbe9c7f562e949b252b82d WatchSource:0}: Error finding container 8544688a7d53af049f463e496d85438a3eb85e844ecbe9c7f562e949b252b82d: Status 404 returned error can't find the container with id 8544688a7d53af049f463e496d85438a3eb85e844ecbe9c7f562e949b252b82d
Sep 10 05:09:06.659238 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:06.659205    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh" event={"ID":"9b9764c5-2d00-4242-8121-1401d22e2fe9","Type":"ContainerStarted","Data":"040ac54a78d90e052b5cd7c04758a5ae7012b0987d680211ee70273cab13f323"}
Sep 10 05:09:06.659613 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:06.659245    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh" event={"ID":"9b9764c5-2d00-4242-8121-1401d22e2fe9","Type":"ContainerStarted","Data":"8544688a7d53af049f463e496d85438a3eb85e844ecbe9c7f562e949b252b82d"}
Sep 10 05:09:06.676746 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:06.676699    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh" podStartSLOduration=1.676683806 podStartE2EDuration="1.676683806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:09:06.674318995 +0000 UTC m=+1683.436088177" watchObservedRunningTime="2026-09-10 05:09:06.676683806 +0000 UTC m=+1683.438452988"
Sep 10 05:09:15.911153 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:15.911120    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:15.911535 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:15.911189    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:15.923830 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:15.923804    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:16.717457 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:16.717422    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:09:38.998112 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:38.997998    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f_83a2d361-102b-464b-8d35-8f3df4f969c8/payload-processing/0.log"
Sep 10 05:09:39.565372 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:39.565331    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:09:39.887344 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:39.887251    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:09:40.061177 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:40.061144    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:09:40.178539 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:40.178436    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:09:42.529242 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:42.529205    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:09:42.627928 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:42.627886    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:09:43.801349 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:43.801311    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:09:43.899209 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:43.899156    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:09:44.138438 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:44.138355    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:09:44.274414 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:44.274375    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:09:44.367948 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:44.367912    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:09:44.407342 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:44.407267    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4"]
Sep 10 05:09:44.410861 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:44.410842    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4"
Sep 10 05:09:44.413476 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:44.413453    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-56fdcd\""
Sep 10 05:09:44.413591 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:44.413539    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-56fdcd-dockercfg-w7rkz\""
Sep 10 05:09:44.422260 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:44.422237    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4"]
Sep 10 05:09:44.455343 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:44.455313    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sfwv\" (UniqueName: \"kubernetes.io/projected/53ea672c-bbcc-4a92-a3a2-6e92d0559777-kube-api-access-2sfwv\") pod \"payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4\" (UID: \"53ea672c-bbcc-4a92-a3a2-6e92d0559777\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4"
Sep 10 05:09:44.455481 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:44.455379    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53ea672c-bbcc-4a92-a3a2-6e92d0559777-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4\" (UID: \"53ea672c-bbcc-4a92-a3a2-6e92d0559777\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4"
Sep 10 05:09:44.556086 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:44.556023    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sfwv\" (UniqueName: \"kubernetes.io/projected/53ea672c-bbcc-4a92-a3a2-6e92d0559777-kube-api-access-2sfwv\") pod \"payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4\" (UID: \"53ea672c-bbcc-4a92-a3a2-6e92d0559777\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4"
Sep 10 05:09:44.556267 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:44.556143    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53ea672c-bbcc-4a92-a3a2-6e92d0559777-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4\" (UID: \"53ea672c-bbcc-4a92-a3a2-6e92d0559777\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4"
Sep 10 05:09:44.556841 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:44.556821    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53ea672c-bbcc-4a92-a3a2-6e92d0559777-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4\" (UID: \"53ea672c-bbcc-4a92-a3a2-6e92d0559777\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4"
Sep 10 05:09:44.564280 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:44.564251    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sfwv\" (UniqueName: \"kubernetes.io/projected/53ea672c-bbcc-4a92-a3a2-6e92d0559777-kube-api-access-2sfwv\") pod \"payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4\" (UID: \"53ea672c-bbcc-4a92-a3a2-6e92d0559777\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4"
Sep 10 05:09:44.722142 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:44.722101    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4"
Sep 10 05:09:44.990723 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:44.990681    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4"]
Sep 10 05:09:44.993260 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:09:44.993228    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ea672c_bbcc_4a92_a3a2_6e92d0559777.slice/crio-1ae72c4f869e838969165e811bfe567d114387bd659ddee44fdc773ebf48bc3c WatchSource:0}: Error finding container 1ae72c4f869e838969165e811bfe567d114387bd659ddee44fdc773ebf48bc3c: Status 404 returned error can't find the container with id 1ae72c4f869e838969165e811bfe567d114387bd659ddee44fdc773ebf48bc3c
Sep 10 05:09:45.848121 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:45.848081    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4" event={"ID":"53ea672c-bbcc-4a92-a3a2-6e92d0559777","Type":"ContainerStarted","Data":"e912e1d1d3e8a957fd52bcadff90a114a4460bb0bb035b28466886e584d27114"}
Sep 10 05:09:45.848299 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:45.848132    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4" event={"ID":"53ea672c-bbcc-4a92-a3a2-6e92d0559777","Type":"ContainerStarted","Data":"1ae72c4f869e838969165e811bfe567d114387bd659ddee44fdc773ebf48bc3c"}
Sep 10 05:09:45.848299 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:45.848156    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4"
Sep 10 05:09:45.864352 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:45.864304    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4" podStartSLOduration=1.864285167 podStartE2EDuration="1.864285167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:09:45.86361584 +0000 UTC m=+1722.625385022" watchObservedRunningTime="2026-09-10 05:09:45.864285167 +0000 UTC m=+1722.626054348"
Sep 10 05:09:56.855241 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:09:56.855211    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4"
Sep 10 05:10:07.649619 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:07.649579    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:10:08.391934 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:08.391896    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56fdcd-mr5sz"]
Sep 10 05:10:08.395355 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:08.395338    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56fdcd-mr5sz"
Sep 10 05:10:08.397938 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:08.397915    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 05:10:08.398083 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:08.397996    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dkf6w\""
Sep 10 05:10:08.403410 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:08.403371    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56fdcd-mr5sz"]
Sep 10 05:10:08.472809 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:08.472769    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5684954c-c858-4a06-b531-ccdbba4fa096-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-56fdcd-mr5sz\" (UID: \"5684954c-c858-4a06-b531-ccdbba4fa096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56fdcd-mr5sz"
Sep 10 05:10:08.573515 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:08.573467    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5684954c-c858-4a06-b531-ccdbba4fa096-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-56fdcd-mr5sz\" (UID: \"5684954c-c858-4a06-b531-ccdbba4fa096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56fdcd-mr5sz"
Sep 10 05:10:08.574237 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:08.574208    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5684954c-c858-4a06-b531-ccdbba4fa096-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-56fdcd-mr5sz\" (UID: \"5684954c-c858-4a06-b531-ccdbba4fa096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56fdcd-mr5sz"
Sep 10 05:10:08.706739 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:08.706709    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56fdcd-mr5sz"
Sep 10 05:10:08.976516 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:08.976488    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56fdcd-mr5sz"]
Sep 10 05:10:08.978572 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:10:08.978548    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5684954c_c858_4a06_b531_ccdbba4fa096.slice/crio-fd557fe956227155255f2d61e8de0e6d820593aebfc0cd503092ca761e039c3b WatchSource:0}: Error finding container fd557fe956227155255f2d61e8de0e6d820593aebfc0cd503092ca761e039c3b: Status 404 returned error can't find the container with id fd557fe956227155255f2d61e8de0e6d820593aebfc0cd503092ca761e039c3b
Sep 10 05:10:09.962228 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:09.962197    2575 generic.go:356] "Generic (PLEG): container finished" podID="5684954c-c858-4a06-b531-ccdbba4fa096" containerID="906c6ac2518ca17c11a2bddbfb652c7d9ca4ace96dd353225fbe3c4c51ac0391" exitCode=0
Sep 10 05:10:09.962597 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:09.962272    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56fdcd-mr5sz" event={"ID":"5684954c-c858-4a06-b531-ccdbba4fa096","Type":"ContainerDied","Data":"906c6ac2518ca17c11a2bddbfb652c7d9ca4ace96dd353225fbe3c4c51ac0391"}
Sep 10 05:10:09.962597 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:09.962341    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56fdcd-mr5sz" event={"ID":"5684954c-c858-4a06-b531-ccdbba4fa096","Type":"ContainerStarted","Data":"fd557fe956227155255f2d61e8de0e6d820593aebfc0cd503092ca761e039c3b"}
Sep 10 05:10:11.098029 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:11.098005    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56fdcd-mr5sz"
Sep 10 05:10:11.195199 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:11.195169    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5684954c-c858-4a06-b531-ccdbba4fa096-maas-api-service-ca\") pod \"5684954c-c858-4a06-b531-ccdbba4fa096\" (UID: \"5684954c-c858-4a06-b531-ccdbba4fa096\") "
Sep 10 05:10:11.195584 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:11.195559    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5684954c-c858-4a06-b531-ccdbba4fa096-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5684954c-c858-4a06-b531-ccdbba4fa096" (UID: "5684954c-c858-4a06-b531-ccdbba4fa096"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:10:11.296255 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:11.296173    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5684954c-c858-4a06-b531-ccdbba4fa096-maas-api-service-ca\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:10:11.972823 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:11.972792    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56fdcd-mr5sz"
Sep 10 05:10:11.972972 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:11.972823    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56fdcd-mr5sz" event={"ID":"5684954c-c858-4a06-b531-ccdbba4fa096","Type":"ContainerDied","Data":"fd557fe956227155255f2d61e8de0e6d820593aebfc0cd503092ca761e039c3b"}
Sep 10 05:10:11.972972 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:11.972875    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd557fe956227155255f2d61e8de0e6d820593aebfc0cd503092ca761e039c3b"
Sep 10 05:10:14.399913 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:14.399871    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:10:14.415946 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:14.415914    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4"]
Sep 10 05:10:14.416157 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:14.416133    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4" podUID="53ea672c-bbcc-4a92-a3a2-6e92d0559777" containerName="payload-processing" containerID="cri-o://e912e1d1d3e8a957fd52bcadff90a114a4460bb0bb035b28466886e584d27114" gracePeriod=30
Sep 10 05:10:14.500719 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:14.500676    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:10:14.600256 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:14.600194    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56fdcd-mr5sz"]
Sep 10 05:10:14.602957 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:14.602921    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:10:14.604012 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:14.603984    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56fdcd-mr5sz"]
Sep 10 05:10:14.666502 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:14.666472    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4"
Sep 10 05:10:14.688653 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:14.688619    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:10:14.829023 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:14.828992    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53ea672c-bbcc-4a92-a3a2-6e92d0559777-plugins-config-volume\") pod \"53ea672c-bbcc-4a92-a3a2-6e92d0559777\" (UID: \"53ea672c-bbcc-4a92-a3a2-6e92d0559777\") "
Sep 10 05:10:14.829310 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:14.829222    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sfwv\" (UniqueName: \"kubernetes.io/projected/53ea672c-bbcc-4a92-a3a2-6e92d0559777-kube-api-access-2sfwv\") pod \"53ea672c-bbcc-4a92-a3a2-6e92d0559777\" (UID: \"53ea672c-bbcc-4a92-a3a2-6e92d0559777\") "
Sep 10 05:10:14.829478 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:14.829450    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ea672c-bbcc-4a92-a3a2-6e92d0559777-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "53ea672c-bbcc-4a92-a3a2-6e92d0559777" (UID: "53ea672c-bbcc-4a92-a3a2-6e92d0559777"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:10:14.831287 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:14.831258    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ea672c-bbcc-4a92-a3a2-6e92d0559777-kube-api-access-2sfwv" (OuterVolumeSpecName: "kube-api-access-2sfwv") pod "53ea672c-bbcc-4a92-a3a2-6e92d0559777" (UID: "53ea672c-bbcc-4a92-a3a2-6e92d0559777"). InnerVolumeSpecName "kube-api-access-2sfwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:10:14.930437 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:14.930352    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sfwv\" (UniqueName: \"kubernetes.io/projected/53ea672c-bbcc-4a92-a3a2-6e92d0559777-kube-api-access-2sfwv\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:10:14.930437 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:14.930384    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53ea672c-bbcc-4a92-a3a2-6e92d0559777-plugins-config-volume\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:10:14.988988 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:14.988953    2575 generic.go:356] "Generic (PLEG): container finished" podID="53ea672c-bbcc-4a92-a3a2-6e92d0559777" containerID="e912e1d1d3e8a957fd52bcadff90a114a4460bb0bb035b28466886e584d27114" exitCode=0
Sep 10 05:10:14.989180 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:14.988999    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4"
Sep 10 05:10:14.989180 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:14.989038    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4" event={"ID":"53ea672c-bbcc-4a92-a3a2-6e92d0559777","Type":"ContainerDied","Data":"e912e1d1d3e8a957fd52bcadff90a114a4460bb0bb035b28466886e584d27114"}
Sep 10 05:10:14.989180 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:14.989111    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4" event={"ID":"53ea672c-bbcc-4a92-a3a2-6e92d0559777","Type":"ContainerDied","Data":"1ae72c4f869e838969165e811bfe567d114387bd659ddee44fdc773ebf48bc3c"}
Sep 10 05:10:14.989180 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:14.989141    2575 scope.go:117] "RemoveContainer" containerID="e912e1d1d3e8a957fd52bcadff90a114a4460bb0bb035b28466886e584d27114"
Sep 10 05:10:14.998268 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:14.998247    2575 scope.go:117] "RemoveContainer" containerID="e912e1d1d3e8a957fd52bcadff90a114a4460bb0bb035b28466886e584d27114"
Sep 10 05:10:14.998529 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:10:14.998510    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e912e1d1d3e8a957fd52bcadff90a114a4460bb0bb035b28466886e584d27114\": container with ID starting with e912e1d1d3e8a957fd52bcadff90a114a4460bb0bb035b28466886e584d27114 not found: ID does not exist" containerID="e912e1d1d3e8a957fd52bcadff90a114a4460bb0bb035b28466886e584d27114"
Sep 10 05:10:14.998581 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:14.998537    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e912e1d1d3e8a957fd52bcadff90a114a4460bb0bb035b28466886e584d27114"} err="failed to get container status \"e912e1d1d3e8a957fd52bcadff90a114a4460bb0bb035b28466886e584d27114\": rpc error: code = NotFound desc = could not find container \"e912e1d1d3e8a957fd52bcadff90a114a4460bb0bb035b28466886e584d27114\": container with ID starting with e912e1d1d3e8a957fd52bcadff90a114a4460bb0bb035b28466886e584d27114 not found: ID does not exist"
Sep 10 05:10:15.005900 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:15.005876    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:10:15.011481 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:15.011458    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4"]
Sep 10 05:10:15.014772 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:15.014748    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-56fdcd-b867c7fc9-ql9c4"]
Sep 10 05:10:15.862235 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:15.862204    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53ea672c-bbcc-4a92-a3a2-6e92d0559777" path="/var/lib/kubelet/pods/53ea672c-bbcc-4a92-a3a2-6e92d0559777/volumes"
Sep 10 05:10:15.862633 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:15.862520    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5684954c-c858-4a06-b531-ccdbba4fa096" path="/var/lib/kubelet/pods/5684954c-c858-4a06-b531-ccdbba4fa096/volumes"
Sep 10 05:10:20.851705 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:20.851655    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:10:20.951825 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:20.951785    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:10:21.161236 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:21.161150    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:10:24.937315 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:24.937278    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95427a-bbg5k"]
Sep 10 05:10:24.937755 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:24.937605    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53ea672c-bbcc-4a92-a3a2-6e92d0559777" containerName="payload-processing"
Sep 10 05:10:24.937755 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:24.937615    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="53ea672c-bbcc-4a92-a3a2-6e92d0559777" containerName="payload-processing"
Sep 10 05:10:24.937755 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:24.937631    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5684954c-c858-4a06-b531-ccdbba4fa096" containerName="revoke-keys"
Sep 10 05:10:24.937755 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:24.937637    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5684954c-c858-4a06-b531-ccdbba4fa096" containerName="revoke-keys"
Sep 10 05:10:24.937755 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:24.937698    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5684954c-c858-4a06-b531-ccdbba4fa096" containerName="revoke-keys"
Sep 10 05:10:24.937755 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:24.937708    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="53ea672c-bbcc-4a92-a3a2-6e92d0559777" containerName="payload-processing"
Sep 10 05:10:24.941913 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:24.941891    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95427a-bbg5k"
Sep 10 05:10:24.944493 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:24.944466    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 05:10:24.944625 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:24.944473    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dkf6w\""
Sep 10 05:10:24.953834 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:24.953809    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95427a-bbg5k"]
Sep 10 05:10:25.014883 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:25.014847    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ac81aa9-70a4-4cd4-97c0-e3af263856d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-95427a-bbg5k\" (UID: \"7ac81aa9-70a4-4cd4-97c0-e3af263856d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95427a-bbg5k"
Sep 10 05:10:25.115384 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:25.115348    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ac81aa9-70a4-4cd4-97c0-e3af263856d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-95427a-bbg5k\" (UID: \"7ac81aa9-70a4-4cd4-97c0-e3af263856d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95427a-bbg5k"
Sep 10 05:10:25.115794 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:25.115761    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:10:25.116030 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:25.116012    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ac81aa9-70a4-4cd4-97c0-e3af263856d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-95427a-bbg5k\" (UID: \"7ac81aa9-70a4-4cd4-97c0-e3af263856d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95427a-bbg5k"
Sep 10 05:10:25.251980 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:25.251948    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95427a-bbg5k"
Sep 10 05:10:25.521775 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:25.521750    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95427a-bbg5k"]
Sep 10 05:10:25.523562 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:10:25.523535    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac81aa9_70a4_4cd4_97c0_e3af263856d2.slice/crio-7ad3c98259ae9496b081511f9f669335e78ec148dd46769a2feb3b8101aebe6b WatchSource:0}: Error finding container 7ad3c98259ae9496b081511f9f669335e78ec148dd46769a2feb3b8101aebe6b: Status 404 returned error can't find the container with id 7ad3c98259ae9496b081511f9f669335e78ec148dd46769a2feb3b8101aebe6b
Sep 10 05:10:26.045930 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:26.045895    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95427a-bbg5k" event={"ID":"7ac81aa9-70a4-4cd4-97c0-e3af263856d2","Type":"ContainerStarted","Data":"8c7f088fc00bcc40e8d1ef92385c14af0d2a90a29c3b64385264896da63303be"}
Sep 10 05:10:26.046323 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:26.045951    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95427a-bbg5k" event={"ID":"7ac81aa9-70a4-4cd4-97c0-e3af263856d2","Type":"ContainerStarted","Data":"7ad3c98259ae9496b081511f9f669335e78ec148dd46769a2feb3b8101aebe6b"}
Sep 10 05:10:26.062203 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:26.062156    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95427a-bbg5k" podStartSLOduration=2.062141733 podStartE2EDuration="2.062141733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:10:26.0605905 +0000 UTC m=+1762.822359694" watchObservedRunningTime="2026-09-10 05:10:26.062141733 +0000 UTC m=+1762.823910946"
Sep 10 05:10:27.052459 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:27.052425    2575 generic.go:356] "Generic (PLEG): container finished" podID="7ac81aa9-70a4-4cd4-97c0-e3af263856d2" containerID="8c7f088fc00bcc40e8d1ef92385c14af0d2a90a29c3b64385264896da63303be" exitCode=0
Sep 10 05:10:27.052861 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:27.052512    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95427a-bbg5k" event={"ID":"7ac81aa9-70a4-4cd4-97c0-e3af263856d2","Type":"ContainerDied","Data":"8c7f088fc00bcc40e8d1ef92385c14af0d2a90a29c3b64385264896da63303be"}
Sep 10 05:10:28.185048 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:28.185022    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95427a-bbg5k"
Sep 10 05:10:28.243198 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:28.243165    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ac81aa9-70a4-4cd4-97c0-e3af263856d2-maas-api-service-ca\") pod \"7ac81aa9-70a4-4cd4-97c0-e3af263856d2\" (UID: \"7ac81aa9-70a4-4cd4-97c0-e3af263856d2\") "
Sep 10 05:10:28.243545 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:28.243523    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ac81aa9-70a4-4cd4-97c0-e3af263856d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7ac81aa9-70a4-4cd4-97c0-e3af263856d2" (UID: "7ac81aa9-70a4-4cd4-97c0-e3af263856d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:10:28.344474 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:28.344377    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ac81aa9-70a4-4cd4-97c0-e3af263856d2-maas-api-service-ca\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:10:29.066277 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:29.066246    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95427a-bbg5k"
Sep 10 05:10:29.066452 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:29.066280    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95427a-bbg5k" event={"ID":"7ac81aa9-70a4-4cd4-97c0-e3af263856d2","Type":"ContainerDied","Data":"7ad3c98259ae9496b081511f9f669335e78ec148dd46769a2feb3b8101aebe6b"}
Sep 10 05:10:29.066452 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:29.066328    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ad3c98259ae9496b081511f9f669335e78ec148dd46769a2feb3b8101aebe6b"
Sep 10 05:10:40.474945 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:40.474899    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:10:40.496877 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:40.496845    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f"]
Sep 10 05:10:40.497165 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:40.497134    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f" podUID="83a2d361-102b-464b-8d35-8f3df4f969c8" containerName="payload-processing" containerID="cri-o://0d9c1a9ffe30517f6be7692cbe28efe37dfe3b0f9f25576ecb54cfbad980ee19" gracePeriod=30
Sep 10 05:10:40.550424 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:40.550392    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:10:40.651516 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:40.651486    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:10:40.674805 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:40.674766    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95427a-bbg5k"]
Sep 10 05:10:40.677867 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:40.677847    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95427a-bbg5k"]
Sep 10 05:10:40.720904 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:40.720870    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:10:40.749423 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:40.749399    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f"
Sep 10 05:10:40.861349 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:40.861321    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a2d361-102b-464b-8d35-8f3df4f969c8-plugins-config-volume\") pod \"83a2d361-102b-464b-8d35-8f3df4f969c8\" (UID: \"83a2d361-102b-464b-8d35-8f3df4f969c8\") "
Sep 10 05:10:40.861508 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:40.861361    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbv2s\" (UniqueName: \"kubernetes.io/projected/83a2d361-102b-464b-8d35-8f3df4f969c8-kube-api-access-wbv2s\") pod \"83a2d361-102b-464b-8d35-8f3df4f969c8\" (UID: \"83a2d361-102b-464b-8d35-8f3df4f969c8\") "
Sep 10 05:10:40.861746 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:40.861721    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a2d361-102b-464b-8d35-8f3df4f969c8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "83a2d361-102b-464b-8d35-8f3df4f969c8" (UID: "83a2d361-102b-464b-8d35-8f3df4f969c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:10:40.863437 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:40.863408    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a2d361-102b-464b-8d35-8f3df4f969c8-kube-api-access-wbv2s" (OuterVolumeSpecName: "kube-api-access-wbv2s") pod "83a2d361-102b-464b-8d35-8f3df4f969c8" (UID: "83a2d361-102b-464b-8d35-8f3df4f969c8"). InnerVolumeSpecName "kube-api-access-wbv2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:10:40.962769 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:40.962736    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a2d361-102b-464b-8d35-8f3df4f969c8-plugins-config-volume\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:10:40.962769 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:40.962761    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbv2s\" (UniqueName: \"kubernetes.io/projected/83a2d361-102b-464b-8d35-8f3df4f969c8-kube-api-access-wbv2s\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:10:41.102138 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:41.102034    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:10:41.124123 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:41.124088    2575 generic.go:356] "Generic (PLEG): container finished" podID="83a2d361-102b-464b-8d35-8f3df4f969c8" containerID="0d9c1a9ffe30517f6be7692cbe28efe37dfe3b0f9f25576ecb54cfbad980ee19" exitCode=0
Sep 10 05:10:41.124348 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:41.124129    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f"
Sep 10 05:10:41.124348 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:41.124153    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f" event={"ID":"83a2d361-102b-464b-8d35-8f3df4f969c8","Type":"ContainerDied","Data":"0d9c1a9ffe30517f6be7692cbe28efe37dfe3b0f9f25576ecb54cfbad980ee19"}
Sep 10 05:10:41.124348 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:41.124203    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f" event={"ID":"83a2d361-102b-464b-8d35-8f3df4f969c8","Type":"ContainerDied","Data":"5c8c6abc5f948f1869a98c5502af5ab389e4f27a72e94438197ea73d83a1ace0"}
Sep 10 05:10:41.124348 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:41.124224    2575 scope.go:117] "RemoveContainer" containerID="0d9c1a9ffe30517f6be7692cbe28efe37dfe3b0f9f25576ecb54cfbad980ee19"
Sep 10 05:10:41.132844 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:41.132826    2575 scope.go:117] "RemoveContainer" containerID="0d9c1a9ffe30517f6be7692cbe28efe37dfe3b0f9f25576ecb54cfbad980ee19"
Sep 10 05:10:41.133134 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:10:41.133114    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9c1a9ffe30517f6be7692cbe28efe37dfe3b0f9f25576ecb54cfbad980ee19\": container with ID starting with 0d9c1a9ffe30517f6be7692cbe28efe37dfe3b0f9f25576ecb54cfbad980ee19 not found: ID does not exist" containerID="0d9c1a9ffe30517f6be7692cbe28efe37dfe3b0f9f25576ecb54cfbad980ee19"
Sep 10 05:10:41.133223 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:41.133142    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9c1a9ffe30517f6be7692cbe28efe37dfe3b0f9f25576ecb54cfbad980ee19"} err="failed to get container status \"0d9c1a9ffe30517f6be7692cbe28efe37dfe3b0f9f25576ecb54cfbad980ee19\": rpc error: code = NotFound desc = could not find container \"0d9c1a9ffe30517f6be7692cbe28efe37dfe3b0f9f25576ecb54cfbad980ee19\": container with ID starting with 0d9c1a9ffe30517f6be7692cbe28efe37dfe3b0f9f25576ecb54cfbad980ee19 not found: ID does not exist"
Sep 10 05:10:41.145873 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:41.145848    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f"]
Sep 10 05:10:41.151610 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:41.151589    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95427a-85cd475c65-grf9f"]
Sep 10 05:10:41.861973 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:41.861939    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ac81aa9-70a4-4cd4-97c0-e3af263856d2" path="/var/lib/kubelet/pods/7ac81aa9-70a4-4cd4-97c0-e3af263856d2/volumes"
Sep 10 05:10:41.862345 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:41.862314    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83a2d361-102b-464b-8d35-8f3df4f969c8" path="/var/lib/kubelet/pods/83a2d361-102b-464b-8d35-8f3df4f969c8/volumes"
Sep 10 05:10:46.082775 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:46.082740    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"]
Sep 10 05:10:46.132096 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:10:46.132034    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:11:01.102216 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:01.102094    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh" podUID="9b9764c5-2d00-4242-8121-1401d22e2fe9" containerName="main" containerID="cri-o://040ac54a78d90e052b5cd7c04758a5ae7012b0987d680211ee70273cab13f323" gracePeriod=45
Sep 10 05:11:01.223339 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:01.223298    2575 generic.go:356] "Generic (PLEG): container finished" podID="9b9764c5-2d00-4242-8121-1401d22e2fe9" containerID="040ac54a78d90e052b5cd7c04758a5ae7012b0987d680211ee70273cab13f323" exitCode=0
Sep 10 05:11:01.223472 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:01.223375    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh" event={"ID":"9b9764c5-2d00-4242-8121-1401d22e2fe9","Type":"ContainerDied","Data":"040ac54a78d90e052b5cd7c04758a5ae7012b0987d680211ee70273cab13f323"}
Sep 10 05:11:01.350037 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:01.350012    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:11:01.446707 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:01.446605    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sng2m\" (UniqueName: \"kubernetes.io/projected/9b9764c5-2d00-4242-8121-1401d22e2fe9-kube-api-access-sng2m\") pod \"9b9764c5-2d00-4242-8121-1401d22e2fe9\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") "
Sep 10 05:11:01.446707 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:01.446647    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-home\") pod \"9b9764c5-2d00-4242-8121-1401d22e2fe9\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") "
Sep 10 05:11:01.446707 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:01.446665    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b9764c5-2d00-4242-8121-1401d22e2fe9-tls-certs\") pod \"9b9764c5-2d00-4242-8121-1401d22e2fe9\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") "
Sep 10 05:11:01.446707 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:01.446686    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-dshm\") pod \"9b9764c5-2d00-4242-8121-1401d22e2fe9\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") "
Sep 10 05:11:01.446707 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:01.446710    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-model-cache\") pod \"9b9764c5-2d00-4242-8121-1401d22e2fe9\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") "
Sep 10 05:11:01.447149 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:01.446739    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-tmp-dir\") pod \"9b9764c5-2d00-4242-8121-1401d22e2fe9\" (UID: \"9b9764c5-2d00-4242-8121-1401d22e2fe9\") "
Sep 10 05:11:01.447149 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:01.446969    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-home" (OuterVolumeSpecName: "home") pod "9b9764c5-2d00-4242-8121-1401d22e2fe9" (UID: "9b9764c5-2d00-4242-8121-1401d22e2fe9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:11:01.447149 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:01.447039    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b9764c5-2d00-4242-8121-1401d22e2fe9" (UID: "9b9764c5-2d00-4242-8121-1401d22e2fe9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:11:01.447308 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:01.447149    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b9764c5-2d00-4242-8121-1401d22e2fe9" (UID: "9b9764c5-2d00-4242-8121-1401d22e2fe9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:11:01.449045 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:01.449005    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b9764c5-2d00-4242-8121-1401d22e2fe9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b9764c5-2d00-4242-8121-1401d22e2fe9" (UID: "9b9764c5-2d00-4242-8121-1401d22e2fe9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 05:11:01.449172 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:01.449012    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-dshm" (OuterVolumeSpecName: "dshm") pod "9b9764c5-2d00-4242-8121-1401d22e2fe9" (UID: "9b9764c5-2d00-4242-8121-1401d22e2fe9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:11:01.449172 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:01.449104    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9764c5-2d00-4242-8121-1401d22e2fe9-kube-api-access-sng2m" (OuterVolumeSpecName: "kube-api-access-sng2m") pod "9b9764c5-2d00-4242-8121-1401d22e2fe9" (UID: "9b9764c5-2d00-4242-8121-1401d22e2fe9"). InnerVolumeSpecName "kube-api-access-sng2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:11:01.547359 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:01.547321    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sng2m\" (UniqueName: \"kubernetes.io/projected/9b9764c5-2d00-4242-8121-1401d22e2fe9-kube-api-access-sng2m\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:11:01.547359 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:01.547353    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-home\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:11:01.547359 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:01.547364    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b9764c5-2d00-4242-8121-1401d22e2fe9-tls-certs\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:11:01.547605 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:01.547374    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-dshm\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:11:01.547605 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:01.547383    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-model-cache\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:11:01.547605 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:01.547412    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b9764c5-2d00-4242-8121-1401d22e2fe9-tmp-dir\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:11:02.230429 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:02.230390    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh" event={"ID":"9b9764c5-2d00-4242-8121-1401d22e2fe9","Type":"ContainerDied","Data":"8544688a7d53af049f463e496d85438a3eb85e844ecbe9c7f562e949b252b82d"}
Sep 10 05:11:02.230904 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:02.230363    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"
Sep 10 05:11:02.230904 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:02.230800    2575 scope.go:117] "RemoveContainer" containerID="040ac54a78d90e052b5cd7c04758a5ae7012b0987d680211ee70273cab13f323"
Sep 10 05:11:02.253867 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:02.253837    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"]
Sep 10 05:11:02.257218 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:02.257192    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-95427a/ipp-route-95427a-kserve-77446c58dc-kpvwh"]
Sep 10 05:11:03.805189 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:03.805161    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2hd2_9d909143-4c70-4fe1-900a-3cd520db32e5/ovn-acl-logging/0.log"
Sep 10 05:11:03.863301 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:03.863271    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b9764c5-2d00-4242-8121-1401d22e2fe9" path="/var/lib/kubelet/pods/9b9764c5-2d00-4242-8121-1401d22e2fe9/volumes"
Sep 10 05:11:03.911541 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:03.911515    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2hd2_9d909143-4c70-4fe1-900a-3cd520db32e5/ovn-acl-logging/0.log"
Sep 10 05:11:22.301559 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:22.301524    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:11:22.965208 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:22.965170    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:11:23.782040 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:23.781999    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:11:28.478280 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:28.478240    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:11:28.543213 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:28.543172    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:11:28.747260 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:28.747157    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:11:29.116125 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:29.116036    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c7644-7zrhc"]
Sep 10 05:11:29.116393 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:29.116380    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83a2d361-102b-464b-8d35-8f3df4f969c8" containerName="payload-processing"
Sep 10 05:11:29.116436 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:29.116394    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="83a2d361-102b-464b-8d35-8f3df4f969c8" containerName="payload-processing"
Sep 10 05:11:29.116436 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:29.116407    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b9764c5-2d00-4242-8121-1401d22e2fe9" containerName="main"
Sep 10 05:11:29.116436 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:29.116412    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b9764c5-2d00-4242-8121-1401d22e2fe9" containerName="main"
Sep 10 05:11:29.116436 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:29.116425    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ac81aa9-70a4-4cd4-97c0-e3af263856d2" containerName="revoke-keys"
Sep 10 05:11:29.116436 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:29.116430    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ac81aa9-70a4-4cd4-97c0-e3af263856d2" containerName="revoke-keys"
Sep 10 05:11:29.116591 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:29.116490    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ac81aa9-70a4-4cd4-97c0-e3af263856d2" containerName="revoke-keys"
Sep 10 05:11:29.116591 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:29.116499    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="83a2d361-102b-464b-8d35-8f3df4f969c8" containerName="payload-processing"
Sep 10 05:11:29.116591 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:29.116506    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b9764c5-2d00-4242-8121-1401d22e2fe9" containerName="main"
Sep 10 05:11:29.119612 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:29.119593    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c7644-7zrhc"
Sep 10 05:11:29.122001 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:29.121967    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 05:11:29.122180 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:29.122094    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dkf6w\""
Sep 10 05:11:29.126169 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:29.126135    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c7644-7zrhc"]
Sep 10 05:11:29.184625 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:29.184592    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5a2038d-19e6-4b57-90b0-68687d7fa032-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1c7644-7zrhc\" (UID: \"e5a2038d-19e6-4b57-90b0-68687d7fa032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c7644-7zrhc"
Sep 10 05:11:29.214147 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:29.214110    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:11:29.285676 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:29.285641    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5a2038d-19e6-4b57-90b0-68687d7fa032-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1c7644-7zrhc\" (UID: \"e5a2038d-19e6-4b57-90b0-68687d7fa032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c7644-7zrhc"
Sep 10 05:11:29.286556 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:29.286536    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5a2038d-19e6-4b57-90b0-68687d7fa032-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1c7644-7zrhc\" (UID: \"e5a2038d-19e6-4b57-90b0-68687d7fa032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c7644-7zrhc"
Sep 10 05:11:29.431138 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:29.431030    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c7644-7zrhc"
Sep 10 05:11:29.702200 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:29.702173    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c7644-7zrhc"]
Sep 10 05:11:30.362361 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:30.362332    2575 generic.go:356] "Generic (PLEG): container finished" podID="e5a2038d-19e6-4b57-90b0-68687d7fa032" containerID="48780071773de783682c1f3c5687d5cb9677678a3c84b15f44462515657bb8f4" exitCode=0
Sep 10 05:11:30.362485 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:30.362420    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c7644-7zrhc" event={"ID":"e5a2038d-19e6-4b57-90b0-68687d7fa032","Type":"ContainerDied","Data":"48780071773de783682c1f3c5687d5cb9677678a3c84b15f44462515657bb8f4"}
Sep 10 05:11:30.362485 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:30.362477    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c7644-7zrhc" event={"ID":"e5a2038d-19e6-4b57-90b0-68687d7fa032","Type":"ContainerStarted","Data":"e27c3c86e3f1c151196c2e02ada5ddfc1636116c5cefa32519480c576fb69bd2"}
Sep 10 05:11:31.494637 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:31.494613    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c7644-7zrhc"
Sep 10 05:11:31.606318 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:31.606288    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5a2038d-19e6-4b57-90b0-68687d7fa032-maas-api-service-ca\") pod \"e5a2038d-19e6-4b57-90b0-68687d7fa032\" (UID: \"e5a2038d-19e6-4b57-90b0-68687d7fa032\") "
Sep 10 05:11:31.606661 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:31.606634    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5a2038d-19e6-4b57-90b0-68687d7fa032-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e5a2038d-19e6-4b57-90b0-68687d7fa032" (UID: "e5a2038d-19e6-4b57-90b0-68687d7fa032"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:11:31.707734 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:31.707700    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5a2038d-19e6-4b57-90b0-68687d7fa032-maas-api-service-ca\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:11:32.374769 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:32.374738    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c7644-7zrhc" event={"ID":"e5a2038d-19e6-4b57-90b0-68687d7fa032","Type":"ContainerDied","Data":"e27c3c86e3f1c151196c2e02ada5ddfc1636116c5cefa32519480c576fb69bd2"}
Sep 10 05:11:32.374769 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:32.374760    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c7644-7zrhc"
Sep 10 05:11:32.374769 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:32.374787    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e27c3c86e3f1c151196c2e02ada5ddfc1636116c5cefa32519480c576fb69bd2"
Sep 10 05:11:39.657850 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:39.657806    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:11:39.724331 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:39.724290    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:11:39.781656 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:39.781610    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:11:39.838412 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:39.838379    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c7644-7zrhc"]
Sep 10 05:11:39.841842 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:39.841813    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c7644-7zrhc"]
Sep 10 05:11:39.842075 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:39.842041    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:11:39.862386 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:39.862358    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5a2038d-19e6-4b57-90b0-68687d7fa032" path="/var/lib/kubelet/pods/e5a2038d-19e6-4b57-90b0-68687d7fa032/volumes"
Sep 10 05:11:39.892250 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:39.892212    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:11:40.241436 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:40.241390    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:11:45.899124 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:45.898962    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:11:45.999720 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:45.999691    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:11:46.168486 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:46.168391    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:11:46.524949 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:46.524915    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4bfef6-587nf"]
Sep 10 05:11:46.525277 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:46.525264    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5a2038d-19e6-4b57-90b0-68687d7fa032" containerName="revoke-keys"
Sep 10 05:11:46.525328 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:46.525278    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5a2038d-19e6-4b57-90b0-68687d7fa032" containerName="revoke-keys"
Sep 10 05:11:46.525365 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:46.525338    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5a2038d-19e6-4b57-90b0-68687d7fa032" containerName="revoke-keys"
Sep 10 05:11:46.528238 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:46.528223    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4bfef6-587nf"
Sep 10 05:11:46.530495 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:46.530466    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dkf6w\""
Sep 10 05:11:46.530621 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:46.530520    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 05:11:46.534870 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:46.534849    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4bfef6-587nf"]
Sep 10 05:11:46.623570 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:46.623539    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:11:46.636657 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:46.636628    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b245575-f3b4-4a32-adb8-15bb9a98b28b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4bfef6-587nf\" (UID: \"5b245575-f3b4-4a32-adb8-15bb9a98b28b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4bfef6-587nf"
Sep 10 05:11:46.737404 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:46.737371    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b245575-f3b4-4a32-adb8-15bb9a98b28b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4bfef6-587nf\" (UID: \"5b245575-f3b4-4a32-adb8-15bb9a98b28b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4bfef6-587nf"
Sep 10 05:11:46.737962 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:46.737944    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b245575-f3b4-4a32-adb8-15bb9a98b28b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4bfef6-587nf\" (UID: \"5b245575-f3b4-4a32-adb8-15bb9a98b28b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4bfef6-587nf"
Sep 10 05:11:46.839495 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:46.839415    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4bfef6-587nf"
Sep 10 05:11:47.110124 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:47.110098    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4bfef6-587nf"]
Sep 10 05:11:47.112001 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:11:47.111973    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b245575_f3b4_4a32_adb8_15bb9a98b28b.slice/crio-bba81a5261776d637bf15a959a24421458bb5e201a9afdbb6d46b67c5d30725f WatchSource:0}: Error finding container bba81a5261776d637bf15a959a24421458bb5e201a9afdbb6d46b67c5d30725f: Status 404 returned error can't find the container with id bba81a5261776d637bf15a959a24421458bb5e201a9afdbb6d46b67c5d30725f
Sep 10 05:11:47.447657 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:47.447619    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4bfef6-587nf" event={"ID":"5b245575-f3b4-4a32-adb8-15bb9a98b28b","Type":"ContainerStarted","Data":"b3c82ff7b43752f4462168c55a38b79f5faccf85d89c85320d1e9f42ff64649d"}
Sep 10 05:11:47.447850 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:47.447670    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4bfef6-587nf" event={"ID":"5b245575-f3b4-4a32-adb8-15bb9a98b28b","Type":"ContainerStarted","Data":"bba81a5261776d637bf15a959a24421458bb5e201a9afdbb6d46b67c5d30725f"}
Sep 10 05:11:47.463057 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:47.463005    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4bfef6-587nf" podStartSLOduration=1.462989036 podStartE2EDuration="1.462989036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:11:47.461335613 +0000 UTC m=+1844.223104805" watchObservedRunningTime="2026-09-10 05:11:47.462989036 +0000 UTC m=+1844.224758219"
Sep 10 05:11:48.453735 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:48.453704    2575 generic.go:356] "Generic (PLEG): container finished" podID="5b245575-f3b4-4a32-adb8-15bb9a98b28b" containerID="b3c82ff7b43752f4462168c55a38b79f5faccf85d89c85320d1e9f42ff64649d" exitCode=0
Sep 10 05:11:48.454163 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:48.453781    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4bfef6-587nf" event={"ID":"5b245575-f3b4-4a32-adb8-15bb9a98b28b","Type":"ContainerDied","Data":"b3c82ff7b43752f4462168c55a38b79f5faccf85d89c85320d1e9f42ff64649d"}
Sep 10 05:11:49.585271 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:49.585239    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4bfef6-587nf"
Sep 10 05:11:49.761928 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:49.761841    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b245575-f3b4-4a32-adb8-15bb9a98b28b-maas-api-service-ca\") pod \"5b245575-f3b4-4a32-adb8-15bb9a98b28b\" (UID: \"5b245575-f3b4-4a32-adb8-15bb9a98b28b\") "
Sep 10 05:11:49.762242 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:49.762219    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b245575-f3b4-4a32-adb8-15bb9a98b28b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b245575-f3b4-4a32-adb8-15bb9a98b28b" (UID: "5b245575-f3b4-4a32-adb8-15bb9a98b28b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:11:49.863243 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:49.863213    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b245575-f3b4-4a32-adb8-15bb9a98b28b-maas-api-service-ca\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:11:50.464534 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:50.464502    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4bfef6-587nf"
Sep 10 05:11:50.464534 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:50.464529    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4bfef6-587nf" event={"ID":"5b245575-f3b4-4a32-adb8-15bb9a98b28b","Type":"ContainerDied","Data":"bba81a5261776d637bf15a959a24421458bb5e201a9afdbb6d46b67c5d30725f"}
Sep 10 05:11:50.464757 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:50.464573    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bba81a5261776d637bf15a959a24421458bb5e201a9afdbb6d46b67c5d30725f"
Sep 10 05:11:57.040657 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.040619    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:11:57.069011 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.068978    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv"]
Sep 10 05:11:57.069321 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.069292    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv" podUID="27db7247-bb70-4335-a59e-cd68d95d643b" containerName="payload-processing" containerID="cri-o://e17a71a28d7e20c64e4c489090fe36cb201d61377a621a8b41cd778842903b90" gracePeriod=30
Sep 10 05:11:57.076330 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.076295    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn"]
Sep 10 05:11:57.076489 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.076470    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn" podUID="491145ab-5023-4559-b7ac-b079dd80dfb7" containerName="payload-pre-processing" containerID="cri-o://94f644c6f2ecb14768f5d5ee52f7c362d5f60270e4ae0e557bf0785f3a360daf" gracePeriod=30
Sep 10 05:11:57.226952 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.226915    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4bfef6-587nf"]
Sep 10 05:11:57.230200 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.230165    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4bfef6-587nf"]
Sep 10 05:11:57.344138 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.344113    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv"
Sep 10 05:11:57.347088 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.347054    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn"
Sep 10 05:11:57.427868 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.427835    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9x8j\" (UniqueName: \"kubernetes.io/projected/27db7247-bb70-4335-a59e-cd68d95d643b-kube-api-access-r9x8j\") pod \"27db7247-bb70-4335-a59e-cd68d95d643b\" (UID: \"27db7247-bb70-4335-a59e-cd68d95d643b\") "
Sep 10 05:11:57.428100 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.427934    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27db7247-bb70-4335-a59e-cd68d95d643b-plugins-config-volume\") pod \"27db7247-bb70-4335-a59e-cd68d95d643b\" (UID: \"27db7247-bb70-4335-a59e-cd68d95d643b\") "
Sep 10 05:11:57.428100 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.427995    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/491145ab-5023-4559-b7ac-b079dd80dfb7-plugins-config-volume\") pod \"491145ab-5023-4559-b7ac-b079dd80dfb7\" (UID: \"491145ab-5023-4559-b7ac-b079dd80dfb7\") "
Sep 10 05:11:57.428100 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.428054    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fl6g\" (UniqueName: \"kubernetes.io/projected/491145ab-5023-4559-b7ac-b079dd80dfb7-kube-api-access-7fl6g\") pod \"491145ab-5023-4559-b7ac-b079dd80dfb7\" (UID: \"491145ab-5023-4559-b7ac-b079dd80dfb7\") "
Sep 10 05:11:57.428601 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.428575    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27db7247-bb70-4335-a59e-cd68d95d643b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "27db7247-bb70-4335-a59e-cd68d95d643b" (UID: "27db7247-bb70-4335-a59e-cd68d95d643b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:11:57.428725 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.428679    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/491145ab-5023-4559-b7ac-b079dd80dfb7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "491145ab-5023-4559-b7ac-b079dd80dfb7" (UID: "491145ab-5023-4559-b7ac-b079dd80dfb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:11:57.430832 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.430787    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27db7247-bb70-4335-a59e-cd68d95d643b-kube-api-access-r9x8j" (OuterVolumeSpecName: "kube-api-access-r9x8j") pod "27db7247-bb70-4335-a59e-cd68d95d643b" (UID: "27db7247-bb70-4335-a59e-cd68d95d643b"). InnerVolumeSpecName "kube-api-access-r9x8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:11:57.430953 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.430870    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/491145ab-5023-4559-b7ac-b079dd80dfb7-kube-api-access-7fl6g" (OuterVolumeSpecName: "kube-api-access-7fl6g") pod "491145ab-5023-4559-b7ac-b079dd80dfb7" (UID: "491145ab-5023-4559-b7ac-b079dd80dfb7"). InnerVolumeSpecName "kube-api-access-7fl6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:11:57.498492 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.498459    2575 generic.go:356] "Generic (PLEG): container finished" podID="27db7247-bb70-4335-a59e-cd68d95d643b" containerID="e17a71a28d7e20c64e4c489090fe36cb201d61377a621a8b41cd778842903b90" exitCode=0
Sep 10 05:11:57.498649 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.498505    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv"
Sep 10 05:11:57.498649 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.498551    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv" event={"ID":"27db7247-bb70-4335-a59e-cd68d95d643b","Type":"ContainerDied","Data":"e17a71a28d7e20c64e4c489090fe36cb201d61377a621a8b41cd778842903b90"}
Sep 10 05:11:57.498649 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.498609    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv" event={"ID":"27db7247-bb70-4335-a59e-cd68d95d643b","Type":"ContainerDied","Data":"887e78ed5b16028af41c227cdda45156fe9feb7cf0edd9d1c65a00b0da9ebf3f"}
Sep 10 05:11:57.498649 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.498639    2575 scope.go:117] "RemoveContainer" containerID="e17a71a28d7e20c64e4c489090fe36cb201d61377a621a8b41cd778842903b90"
Sep 10 05:11:57.499893 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.499870    2575 generic.go:356] "Generic (PLEG): container finished" podID="491145ab-5023-4559-b7ac-b079dd80dfb7" containerID="94f644c6f2ecb14768f5d5ee52f7c362d5f60270e4ae0e557bf0785f3a360daf" exitCode=0
Sep 10 05:11:57.499973 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.499904    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn"
Sep 10 05:11:57.499973 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.499959    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn" event={"ID":"491145ab-5023-4559-b7ac-b079dd80dfb7","Type":"ContainerDied","Data":"94f644c6f2ecb14768f5d5ee52f7c362d5f60270e4ae0e557bf0785f3a360daf"}
Sep 10 05:11:57.500049 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.500012    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn" event={"ID":"491145ab-5023-4559-b7ac-b079dd80dfb7","Type":"ContainerDied","Data":"b9707f8a38d5a56956a02f81f331b9139376dd5214a50880d8a1340f6c88d006"}
Sep 10 05:11:57.507923 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.507906    2575 scope.go:117] "RemoveContainer" containerID="e17a71a28d7e20c64e4c489090fe36cb201d61377a621a8b41cd778842903b90"
Sep 10 05:11:57.508248 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:11:57.508230    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e17a71a28d7e20c64e4c489090fe36cb201d61377a621a8b41cd778842903b90\": container with ID starting with e17a71a28d7e20c64e4c489090fe36cb201d61377a621a8b41cd778842903b90 not found: ID does not exist" containerID="e17a71a28d7e20c64e4c489090fe36cb201d61377a621a8b41cd778842903b90"
Sep 10 05:11:57.508298 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.508257    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e17a71a28d7e20c64e4c489090fe36cb201d61377a621a8b41cd778842903b90"} err="failed to get container status \"e17a71a28d7e20c64e4c489090fe36cb201d61377a621a8b41cd778842903b90\": rpc error: code = NotFound desc = could not find container \"e17a71a28d7e20c64e4c489090fe36cb201d61377a621a8b41cd778842903b90\": container with ID starting with e17a71a28d7e20c64e4c489090fe36cb201d61377a621a8b41cd778842903b90 not found: ID does not exist"
Sep 10 05:11:57.508298 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.508274    2575 scope.go:117] "RemoveContainer" containerID="94f644c6f2ecb14768f5d5ee52f7c362d5f60270e4ae0e557bf0785f3a360daf"
Sep 10 05:11:57.515931 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.515916    2575 scope.go:117] "RemoveContainer" containerID="94f644c6f2ecb14768f5d5ee52f7c362d5f60270e4ae0e557bf0785f3a360daf"
Sep 10 05:11:57.516210 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:11:57.516193    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94f644c6f2ecb14768f5d5ee52f7c362d5f60270e4ae0e557bf0785f3a360daf\": container with ID starting with 94f644c6f2ecb14768f5d5ee52f7c362d5f60270e4ae0e557bf0785f3a360daf not found: ID does not exist" containerID="94f644c6f2ecb14768f5d5ee52f7c362d5f60270e4ae0e557bf0785f3a360daf"
Sep 10 05:11:57.516264 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.516220    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94f644c6f2ecb14768f5d5ee52f7c362d5f60270e4ae0e557bf0785f3a360daf"} err="failed to get container status \"94f644c6f2ecb14768f5d5ee52f7c362d5f60270e4ae0e557bf0785f3a360daf\": rpc error: code = NotFound desc = could not find container \"94f644c6f2ecb14768f5d5ee52f7c362d5f60270e4ae0e557bf0785f3a360daf\": container with ID starting with 94f644c6f2ecb14768f5d5ee52f7c362d5f60270e4ae0e557bf0785f3a360daf not found: ID does not exist"
Sep 10 05:11:57.523174 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.523152    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn"]
Sep 10 05:11:57.525704 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.525684    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4bfef6-6c87469954-fl6zn"]
Sep 10 05:11:57.528913 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.528898    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fl6g\" (UniqueName: \"kubernetes.io/projected/491145ab-5023-4559-b7ac-b079dd80dfb7-kube-api-access-7fl6g\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:11:57.528987 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.528915    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9x8j\" (UniqueName: \"kubernetes.io/projected/27db7247-bb70-4335-a59e-cd68d95d643b-kube-api-access-r9x8j\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:11:57.528987 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.528925    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27db7247-bb70-4335-a59e-cd68d95d643b-plugins-config-volume\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:11:57.528987 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.528935    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/491145ab-5023-4559-b7ac-b079dd80dfb7-plugins-config-volume\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:11:57.536131 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.536112    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv"]
Sep 10 05:11:57.539480 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.539462    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4bfef6-7784594f89-2g5dv"]
Sep 10 05:11:57.863789 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.863752    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27db7247-bb70-4335-a59e-cd68d95d643b" path="/var/lib/kubelet/pods/27db7247-bb70-4335-a59e-cd68d95d643b/volumes"
Sep 10 05:11:57.864349 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.864326    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="491145ab-5023-4559-b7ac-b079dd80dfb7" path="/var/lib/kubelet/pods/491145ab-5023-4559-b7ac-b079dd80dfb7/volumes"
Sep 10 05:11:57.865035 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:57.865013    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b245575-f3b4-4a32-adb8-15bb9a98b28b" path="/var/lib/kubelet/pods/5b245575-f3b4-4a32-adb8-15bb9a98b28b/volumes"
Sep 10 05:11:58.351091 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:11:58.351022    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:12:03.434299 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:03.434248    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:12:03.490978 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:03.490921    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:12:03.698807 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:03.698770    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:12:04.010599 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:04.010510    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:12:04.061976 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:04.061941    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b702d-xnkkf"]
Sep 10 05:12:04.062301 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:04.062287    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="491145ab-5023-4559-b7ac-b079dd80dfb7" containerName="payload-pre-processing"
Sep 10 05:12:04.062348 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:04.062303    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="491145ab-5023-4559-b7ac-b079dd80dfb7" containerName="payload-pre-processing"
Sep 10 05:12:04.062348 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:04.062324    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b245575-f3b4-4a32-adb8-15bb9a98b28b" containerName="revoke-keys"
Sep 10 05:12:04.062348 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:04.062331    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b245575-f3b4-4a32-adb8-15bb9a98b28b" containerName="revoke-keys"
Sep 10 05:12:04.062348 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:04.062338    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27db7247-bb70-4335-a59e-cd68d95d643b" containerName="payload-processing"
Sep 10 05:12:04.062348 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:04.062344    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="27db7247-bb70-4335-a59e-cd68d95d643b" containerName="payload-processing"
Sep 10 05:12:04.062503 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:04.062407    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b245575-f3b4-4a32-adb8-15bb9a98b28b" containerName="revoke-keys"
Sep 10 05:12:04.062503 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:04.062419    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="491145ab-5023-4559-b7ac-b079dd80dfb7" containerName="payload-pre-processing"
Sep 10 05:12:04.062503 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:04.062426    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="27db7247-bb70-4335-a59e-cd68d95d643b" containerName="payload-processing"
Sep 10 05:12:04.066994 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:04.066974    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b702d-xnkkf"
Sep 10 05:12:04.069678 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:04.069647    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dkf6w\""
Sep 10 05:12:04.069816 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:04.069647    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 05:12:04.073340 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:04.073316    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b702d-xnkkf"]
Sep 10 05:12:04.088045 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:04.088019    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e6ca10-ec0a-4983-bdb7-464ab864912e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3b702d-xnkkf\" (UID: \"02e6ca10-ec0a-4983-bdb7-464ab864912e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b702d-xnkkf"
Sep 10 05:12:04.188480 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:04.188442    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e6ca10-ec0a-4983-bdb7-464ab864912e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3b702d-xnkkf\" (UID: \"02e6ca10-ec0a-4983-bdb7-464ab864912e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b702d-xnkkf"
Sep 10 05:12:04.189250 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:04.189227    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e6ca10-ec0a-4983-bdb7-464ab864912e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3b702d-xnkkf\" (UID: \"02e6ca10-ec0a-4983-bdb7-464ab864912e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b702d-xnkkf"
Sep 10 05:12:04.378560 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:04.378521    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b702d-xnkkf"
Sep 10 05:12:04.647541 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:04.647418    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b702d-xnkkf"]
Sep 10 05:12:04.649777 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:12:04.649745    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e6ca10_ec0a_4983_bdb7_464ab864912e.slice/crio-4fde25f32d9f124b8b93a08a8f65a9c1fd97a2f90766b6dd0d667a3c8f0a5cfd WatchSource:0}: Error finding container 4fde25f32d9f124b8b93a08a8f65a9c1fd97a2f90766b6dd0d667a3c8f0a5cfd: Status 404 returned error can't find the container with id 4fde25f32d9f124b8b93a08a8f65a9c1fd97a2f90766b6dd0d667a3c8f0a5cfd
Sep 10 05:12:04.662389 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:04.662361    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:12:05.544707 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:05.544664    2575 generic.go:356] "Generic (PLEG): container finished" podID="02e6ca10-ec0a-4983-bdb7-464ab864912e" containerID="63ca5342046520949a3871d2af2be13ba3d4f82eeb05f0daee5ebf71ece9ff56" exitCode=0
Sep 10 05:12:05.544877 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:05.544748    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b702d-xnkkf" event={"ID":"02e6ca10-ec0a-4983-bdb7-464ab864912e","Type":"ContainerDied","Data":"63ca5342046520949a3871d2af2be13ba3d4f82eeb05f0daee5ebf71ece9ff56"}
Sep 10 05:12:05.544877 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:05.544800    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b702d-xnkkf" event={"ID":"02e6ca10-ec0a-4983-bdb7-464ab864912e","Type":"ContainerStarted","Data":"4fde25f32d9f124b8b93a08a8f65a9c1fd97a2f90766b6dd0d667a3c8f0a5cfd"}
Sep 10 05:12:06.672137 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:06.672110    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b702d-xnkkf"
Sep 10 05:12:06.710668 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:06.710632    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e6ca10-ec0a-4983-bdb7-464ab864912e-maas-api-service-ca\") pod \"02e6ca10-ec0a-4983-bdb7-464ab864912e\" (UID: \"02e6ca10-ec0a-4983-bdb7-464ab864912e\") "
Sep 10 05:12:06.711012 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:06.710987    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e6ca10-ec0a-4983-bdb7-464ab864912e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "02e6ca10-ec0a-4983-bdb7-464ab864912e" (UID: "02e6ca10-ec0a-4983-bdb7-464ab864912e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:12:06.812274 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:06.812182    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e6ca10-ec0a-4983-bdb7-464ab864912e-maas-api-service-ca\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:12:07.556332 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:07.556292    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b702d-xnkkf" event={"ID":"02e6ca10-ec0a-4983-bdb7-464ab864912e","Type":"ContainerDied","Data":"4fde25f32d9f124b8b93a08a8f65a9c1fd97a2f90766b6dd0d667a3c8f0a5cfd"}
Sep 10 05:12:07.556507 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:07.556345    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fde25f32d9f124b8b93a08a8f65a9c1fd97a2f90766b6dd0d667a3c8f0a5cfd"
Sep 10 05:12:07.556507 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:07.556305    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b702d-xnkkf"
Sep 10 05:12:14.629851 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:14.629816    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:12:14.655719 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:14.655688    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf"]
Sep 10 05:12:14.655971 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:14.655942    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf" podUID="b62e0c61-13cd-45a3-9129-9782cdd02a0b" containerName="payload-pre-processing" containerID="cri-o://1468c72a835383cb4ba454a2b6e583c26cb18d1dc26c51c74c4c28229e4c8203" gracePeriod=30
Sep 10 05:12:14.729790 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:14.729754    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:12:14.799518 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:14.799458    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b702d-xnkkf"]
Sep 10 05:12:14.801520 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:14.801453    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b702d-xnkkf"]
Sep 10 05:12:14.801520 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:14.801482    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:12:14.856317 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:14.856280    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:12:14.922405 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:14.922383    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf"
Sep 10 05:12:14.976452 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:14.976420    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28zgx\" (UniqueName: \"kubernetes.io/projected/b62e0c61-13cd-45a3-9129-9782cdd02a0b-kube-api-access-28zgx\") pod \"b62e0c61-13cd-45a3-9129-9782cdd02a0b\" (UID: \"b62e0c61-13cd-45a3-9129-9782cdd02a0b\") "
Sep 10 05:12:14.976616 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:14.976471    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b62e0c61-13cd-45a3-9129-9782cdd02a0b-plugins-config-volume\") pod \"b62e0c61-13cd-45a3-9129-9782cdd02a0b\" (UID: \"b62e0c61-13cd-45a3-9129-9782cdd02a0b\") "
Sep 10 05:12:14.976812 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:14.976786    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b62e0c61-13cd-45a3-9129-9782cdd02a0b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b62e0c61-13cd-45a3-9129-9782cdd02a0b" (UID: "b62e0c61-13cd-45a3-9129-9782cdd02a0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 05:12:14.978539 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:14.978519    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b62e0c61-13cd-45a3-9129-9782cdd02a0b-kube-api-access-28zgx" (OuterVolumeSpecName: "kube-api-access-28zgx") pod "b62e0c61-13cd-45a3-9129-9782cdd02a0b" (UID: "b62e0c61-13cd-45a3-9129-9782cdd02a0b"). InnerVolumeSpecName "kube-api-access-28zgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:12:15.077991 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:15.077959    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b62e0c61-13cd-45a3-9129-9782cdd02a0b-plugins-config-volume\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:12:15.077991 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:15.077985    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28zgx\" (UniqueName: \"kubernetes.io/projected/b62e0c61-13cd-45a3-9129-9782cdd02a0b-kube-api-access-28zgx\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:12:15.201441 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:15.201403    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:12:15.595047 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:15.594967    2575 generic.go:356] "Generic (PLEG): container finished" podID="b62e0c61-13cd-45a3-9129-9782cdd02a0b" containerID="1468c72a835383cb4ba454a2b6e583c26cb18d1dc26c51c74c4c28229e4c8203" exitCode=0
Sep 10 05:12:15.595047 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:15.595012    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf"
Sep 10 05:12:15.595231 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:15.595055    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf" event={"ID":"b62e0c61-13cd-45a3-9129-9782cdd02a0b","Type":"ContainerDied","Data":"1468c72a835383cb4ba454a2b6e583c26cb18d1dc26c51c74c4c28229e4c8203"}
Sep 10 05:12:15.595231 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:15.595135    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf" event={"ID":"b62e0c61-13cd-45a3-9129-9782cdd02a0b","Type":"ContainerDied","Data":"1817b6c48fa5d903a1b2f4f5c2ea4e4a14ed28427371580445fa4c698fdf956b"}
Sep 10 05:12:15.595231 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:15.595163    2575 scope.go:117] "RemoveContainer" containerID="1468c72a835383cb4ba454a2b6e583c26cb18d1dc26c51c74c4c28229e4c8203"
Sep 10 05:12:15.604529 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:15.604511    2575 scope.go:117] "RemoveContainer" containerID="1468c72a835383cb4ba454a2b6e583c26cb18d1dc26c51c74c4c28229e4c8203"
Sep 10 05:12:15.604820 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:12:15.604798    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1468c72a835383cb4ba454a2b6e583c26cb18d1dc26c51c74c4c28229e4c8203\": container with ID starting with 1468c72a835383cb4ba454a2b6e583c26cb18d1dc26c51c74c4c28229e4c8203 not found: ID does not exist" containerID="1468c72a835383cb4ba454a2b6e583c26cb18d1dc26c51c74c4c28229e4c8203"
Sep 10 05:12:15.604856 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:15.604830    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1468c72a835383cb4ba454a2b6e583c26cb18d1dc26c51c74c4c28229e4c8203"} err="failed to get container status \"1468c72a835383cb4ba454a2b6e583c26cb18d1dc26c51c74c4c28229e4c8203\": rpc error: code = NotFound desc = could not find container \"1468c72a835383cb4ba454a2b6e583c26cb18d1dc26c51c74c4c28229e4c8203\": container with ID starting with 1468c72a835383cb4ba454a2b6e583c26cb18d1dc26c51c74c4c28229e4c8203 not found: ID does not exist"
Sep 10 05:12:15.617384 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:15.617363    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf"]
Sep 10 05:12:15.622892 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:15.622869    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b702d-d96f89998-2pscf"]
Sep 10 05:12:15.862533 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:15.862453    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02e6ca10-ec0a-4983-bdb7-464ab864912e" path="/var/lib/kubelet/pods/02e6ca10-ec0a-4983-bdb7-464ab864912e/volumes"
Sep 10 05:12:15.862859 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:15.862808    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b62e0c61-13cd-45a3-9129-9782cdd02a0b" path="/var/lib/kubelet/pods/b62e0c61-13cd-45a3-9129-9782cdd02a0b/volumes"
Sep 10 05:12:21.124140 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:21.124100    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:12:21.184600 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:21.184564    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:12:21.461680 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:12:21.461649    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:13:04.274945 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:13:04.274912    2575 scope.go:117] "RemoveContainer" containerID="855c31d89a68b6294003780f40ce54538a6cf528fc9e6b91b8f1b05cdb47a27c"
Sep 10 05:13:04.282907 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:13:04.282889    2575 scope.go:117] "RemoveContainer" containerID="4e589f1d97342389780024e620f22cdb7272ab70a0699628c6b3417afca1cf53"
Sep 10 05:13:04.291891 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:13:04.291864    2575 scope.go:117] "RemoveContainer" containerID="9da88d016c758f97bd7b939b3f9260d31b291edeb7881195a16b5e7b4f533c03"
Sep 10 05:13:32.099368 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:13:32.099336    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:13:45.779884 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:13:45.779845    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:13:45.984127 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:13:45.984060    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:13:48.015081 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:13:48.015033    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:14:04.322334 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:14:04.322252    2575 scope.go:117] "RemoveContainer" containerID="0890b387103175094b576d4e43c0a60783fabf0e559a4ec4f760fb6388bdb589"
Sep 10 05:15:10.398369 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:15:10.398332    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:15:10.454546 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:15:10.454511    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:15:35.883427 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:15:35.883334    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:15:35.988642 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:15:35.988601    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:16:03.243357 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:03.243323    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:16:03.830987 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:03.830961    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2hd2_9d909143-4c70-4fe1-900a-3cd520db32e5/ovn-acl-logging/0.log"
Sep 10 05:16:03.936878 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:03.936852    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2hd2_9d909143-4c70-4fe1-900a-3cd520db32e5/ovn-acl-logging/0.log"
Sep 10 05:16:38.095274 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.095234    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"]
Sep 10 05:16:38.095733 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.095703    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b62e0c61-13cd-45a3-9129-9782cdd02a0b" containerName="payload-pre-processing"
Sep 10 05:16:38.095733 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.095719    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b62e0c61-13cd-45a3-9129-9782cdd02a0b" containerName="payload-pre-processing"
Sep 10 05:16:38.095733 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.095732    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e6ca10-ec0a-4983-bdb7-464ab864912e" containerName="revoke-keys"
Sep 10 05:16:38.095841 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.095742    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e6ca10-ec0a-4983-bdb7-464ab864912e" containerName="revoke-keys"
Sep 10 05:16:38.095841 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.095833    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b62e0c61-13cd-45a3-9129-9782cdd02a0b" containerName="payload-pre-processing"
Sep 10 05:16:38.095905 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.095844    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e6ca10-ec0a-4983-bdb7-464ab864912e" containerName="revoke-keys"
Sep 10 05:16:38.098973 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.098957    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:38.101312 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.101289    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-4d897bf1-kserve-self-signed-certs\""
Sep 10 05:16:38.104723 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.104661    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"]
Sep 10 05:16:38.118605 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.118572    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:16:38.185683 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.185642    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-model-cache\") pod \"e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") " pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:38.185683 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.185684    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-tmp-dir\") pod \"e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") " pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:38.185916 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.185769    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-dshm\") pod \"e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") " pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:38.185916 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.185798    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/200688a3-0725-4992-9ec2-176b0afa8fc1-tls-certs\") pod \"e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") " pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:38.185916 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.185834    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-home\") pod \"e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") " pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:38.185916 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.185857    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc7hn\" (UniqueName: \"kubernetes.io/projected/200688a3-0725-4992-9ec2-176b0afa8fc1-kube-api-access-nc7hn\") pod \"e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") " pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:38.286472 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.286414    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-model-cache\") pod \"e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") " pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:38.286472 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.286475    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-tmp-dir\") pod \"e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") " pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:38.286706 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.286552    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-dshm\") pod \"e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") " pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:38.286706 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.286579    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/200688a3-0725-4992-9ec2-176b0afa8fc1-tls-certs\") pod \"e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") " pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:38.286706 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.286620    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-home\") pod \"e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") " pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:38.286706 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.286645    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc7hn\" (UniqueName: \"kubernetes.io/projected/200688a3-0725-4992-9ec2-176b0afa8fc1-kube-api-access-nc7hn\") pod \"e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") " pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:38.286926 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.286908    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-model-cache\") pod \"e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") " pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:38.286990 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.286936    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-tmp-dir\") pod \"e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") " pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:38.287040 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.286988    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-home\") pod \"e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") " pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:38.288904 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.288878    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-dshm\") pod \"e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") " pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:38.289094 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.289053    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/200688a3-0725-4992-9ec2-176b0afa8fc1-tls-certs\") pod \"e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") " pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:38.293872 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.293837    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc7hn\" (UniqueName: \"kubernetes.io/projected/200688a3-0725-4992-9ec2-176b0afa8fc1-kube-api-access-nc7hn\") pod \"e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") " pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:38.409049 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.408958    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:38.676293 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.676267    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"]
Sep 10 05:16:38.678711 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:16:38.678685    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod200688a3_0725_4992_9ec2_176b0afa8fc1.slice/crio-e6132085dc06f2c452e20504ed8815c04d2ed2b104095184d8aca261e97a587a WatchSource:0}: Error finding container e6132085dc06f2c452e20504ed8815c04d2ed2b104095184d8aca261e97a587a: Status 404 returned error can't find the container with id e6132085dc06f2c452e20504ed8815c04d2ed2b104095184d8aca261e97a587a
Sep 10 05:16:38.797926 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.797887    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr" event={"ID":"200688a3-0725-4992-9ec2-176b0afa8fc1","Type":"ContainerStarted","Data":"c67d086bc3cc348ba56e42b164d0e38537f09a2a5ed106402826be1722b517b0"}
Sep 10 05:16:38.798120 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.797942    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr" event={"ID":"200688a3-0725-4992-9ec2-176b0afa8fc1","Type":"ContainerStarted","Data":"e6132085dc06f2c452e20504ed8815c04d2ed2b104095184d8aca261e97a587a"}
Sep 10 05:16:38.813307 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:38.813245    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr" podStartSLOduration=0.813229596 podStartE2EDuration="813.229596ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:16:38.812476745 +0000 UTC m=+2135.574245927" watchObservedRunningTime="2026-09-10 05:16:38.813229596 +0000 UTC m=+2135.574998796"
Sep 10 05:16:48.409516 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:48.409478    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:48.409516 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:48.409523    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:48.421987 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:48.421957    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:16:48.854514 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:16:48.854478    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:17:04.352933 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:17:04.352852    2575 scope.go:117] "RemoveContainer" containerID="906c6ac2518ca17c11a2bddbfb652c7d9ca4ace96dd353225fbe3c4c51ac0391"
Sep 10 05:17:04.361831 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:17:04.361811    2575 scope.go:117] "RemoveContainer" containerID="8c7f088fc00bcc40e8d1ef92385c14af0d2a90a29c3b64385264896da63303be"
Sep 10 05:17:06.529803 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:17:06.529755    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:17:45.063284 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:17:45.063249    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"]
Sep 10 05:17:45.081205 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:17:45.081170    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:17:46.707137 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:17:46.707055    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:18:00.083492 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:00.083429    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr" podUID="200688a3-0725-4992-9ec2-176b0afa8fc1" containerName="main" containerID="cri-o://c67d086bc3cc348ba56e42b164d0e38537f09a2a5ed106402826be1722b517b0" gracePeriod=45
Sep 10 05:18:00.330890 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:00.330866    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:18:00.482970 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:00.482941    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nc7hn\" (UniqueName: \"kubernetes.io/projected/200688a3-0725-4992-9ec2-176b0afa8fc1-kube-api-access-nc7hn\") pod \"200688a3-0725-4992-9ec2-176b0afa8fc1\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") "
Sep 10 05:18:00.483152 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:00.483022    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-dshm\") pod \"200688a3-0725-4992-9ec2-176b0afa8fc1\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") "
Sep 10 05:18:00.483152 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:00.483055    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-model-cache\") pod \"200688a3-0725-4992-9ec2-176b0afa8fc1\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") "
Sep 10 05:18:00.483152 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:00.483099    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/200688a3-0725-4992-9ec2-176b0afa8fc1-tls-certs\") pod \"200688a3-0725-4992-9ec2-176b0afa8fc1\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") "
Sep 10 05:18:00.483152 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:00.483129    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-tmp-dir\") pod \"200688a3-0725-4992-9ec2-176b0afa8fc1\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") "
Sep 10 05:18:00.483337 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:00.483154    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-home\") pod \"200688a3-0725-4992-9ec2-176b0afa8fc1\" (UID: \"200688a3-0725-4992-9ec2-176b0afa8fc1\") "
Sep 10 05:18:00.483396 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:00.483350    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-model-cache" (OuterVolumeSpecName: "model-cache") pod "200688a3-0725-4992-9ec2-176b0afa8fc1" (UID: "200688a3-0725-4992-9ec2-176b0afa8fc1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:18:00.483456 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:00.483413    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "200688a3-0725-4992-9ec2-176b0afa8fc1" (UID: "200688a3-0725-4992-9ec2-176b0afa8fc1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:18:00.483529 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:00.483503    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-home" (OuterVolumeSpecName: "home") pod "200688a3-0725-4992-9ec2-176b0afa8fc1" (UID: "200688a3-0725-4992-9ec2-176b0afa8fc1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:18:00.485407 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:00.485373    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-dshm" (OuterVolumeSpecName: "dshm") pod "200688a3-0725-4992-9ec2-176b0afa8fc1" (UID: "200688a3-0725-4992-9ec2-176b0afa8fc1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 05:18:00.485515 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:00.485406    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/200688a3-0725-4992-9ec2-176b0afa8fc1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "200688a3-0725-4992-9ec2-176b0afa8fc1" (UID: "200688a3-0725-4992-9ec2-176b0afa8fc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 05:18:00.485515 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:00.485453    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/200688a3-0725-4992-9ec2-176b0afa8fc1-kube-api-access-nc7hn" (OuterVolumeSpecName: "kube-api-access-nc7hn") pod "200688a3-0725-4992-9ec2-176b0afa8fc1" (UID: "200688a3-0725-4992-9ec2-176b0afa8fc1"). InnerVolumeSpecName "kube-api-access-nc7hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 05:18:00.584724 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:00.584693    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-dshm\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:18:00.584724 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:00.584719    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-model-cache\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:18:00.584906 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:00.584736    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/200688a3-0725-4992-9ec2-176b0afa8fc1-tls-certs\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:18:00.584906 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:00.584747    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-tmp-dir\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:18:00.584906 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:00.584757    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/200688a3-0725-4992-9ec2-176b0afa8fc1-home\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:18:00.584906 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:00.584768    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc7hn\" (UniqueName: \"kubernetes.io/projected/200688a3-0725-4992-9ec2-176b0afa8fc1-kube-api-access-nc7hn\") on node \"ip-10-0-132-70.ec2.internal\" DevicePath \"\""
Sep 10 05:18:01.163861 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:01.163821    2575 generic.go:356] "Generic (PLEG): container finished" podID="200688a3-0725-4992-9ec2-176b0afa8fc1" containerID="c67d086bc3cc348ba56e42b164d0e38537f09a2a5ed106402826be1722b517b0" exitCode=0
Sep 10 05:18:01.164357 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:01.163893    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"
Sep 10 05:18:01.164357 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:01.163903    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr" event={"ID":"200688a3-0725-4992-9ec2-176b0afa8fc1","Type":"ContainerDied","Data":"c67d086bc3cc348ba56e42b164d0e38537f09a2a5ed106402826be1722b517b0"}
Sep 10 05:18:01.164357 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:01.163955    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr" event={"ID":"200688a3-0725-4992-9ec2-176b0afa8fc1","Type":"ContainerDied","Data":"e6132085dc06f2c452e20504ed8815c04d2ed2b104095184d8aca261e97a587a"}
Sep 10 05:18:01.164357 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:01.163975    2575 scope.go:117] "RemoveContainer" containerID="c67d086bc3cc348ba56e42b164d0e38537f09a2a5ed106402826be1722b517b0"
Sep 10 05:18:01.173973 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:01.173953    2575 scope.go:117] "RemoveContainer" containerID="c67d086bc3cc348ba56e42b164d0e38537f09a2a5ed106402826be1722b517b0"
Sep 10 05:18:01.174306 ip-10-0-132-70 kubenswrapper[2575]: E0910 05:18:01.174281    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c67d086bc3cc348ba56e42b164d0e38537f09a2a5ed106402826be1722b517b0\": container with ID starting with c67d086bc3cc348ba56e42b164d0e38537f09a2a5ed106402826be1722b517b0 not found: ID does not exist" containerID="c67d086bc3cc348ba56e42b164d0e38537f09a2a5ed106402826be1722b517b0"
Sep 10 05:18:01.174401 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:01.174315    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67d086bc3cc348ba56e42b164d0e38537f09a2a5ed106402826be1722b517b0"} err="failed to get container status \"c67d086bc3cc348ba56e42b164d0e38537f09a2a5ed106402826be1722b517b0\": rpc error: code = NotFound desc = could not find container \"c67d086bc3cc348ba56e42b164d0e38537f09a2a5ed106402826be1722b517b0\": container with ID starting with c67d086bc3cc348ba56e42b164d0e38537f09a2a5ed106402826be1722b517b0 not found: ID does not exist"
Sep 10 05:18:01.186214 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:01.186182    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"]
Sep 10 05:18:01.188983 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:01.188955    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-4d897bf1-kserve-54ff76f4dc-8jhjr"]
Sep 10 05:18:01.863252 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:01.863217    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="200688a3-0725-4992-9ec2-176b0afa8fc1" path="/var/lib/kubelet/pods/200688a3-0725-4992-9ec2-176b0afa8fc1/volumes"
Sep 10 05:18:04.381083 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:04.381043    2575 scope.go:117] "RemoveContainer" containerID="b3c82ff7b43752f4462168c55a38b79f5faccf85d89c85320d1e9f42ff64649d"
Sep 10 05:18:04.388746 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:04.388722    2575 scope.go:117] "RemoveContainer" containerID="48780071773de783682c1f3c5687d5cb9677678a3c84b15f44462515657bb8f4"
Sep 10 05:18:40.875718 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:40.875629    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78fc98c8bf-kpwwf_a336d026-9d0e-4ac6-81d8-08e992654b98/authorino/0.log"
Sep 10 05:18:46.494126 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:46.494094    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-6r8nx_2bfb57ba-1de6-45ef-9e88-28d442ad2131/server/0.log"
Sep 10 05:18:46.614271 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:46.614190    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-jxfkk_822d9fab-f302-4685-93e1-ef41d09cfd1a/manager/0.log"
Sep 10 05:18:46.979825 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:46.979798    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c57f4866c-qztwq_8ee33fc9-f1e7-4b32-af9f-d3a724d2760e/copy-manifests/0.log"
Sep 10 05:18:46.996804 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:46.996769    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c57f4866c-qztwq_8ee33fc9-f1e7-4b32-af9f-d3a724d2760e/manager/0.log"
Sep 10 05:18:47.581898 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:47.581865    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29816940-dfrgf_5c00d475-da16-4e81-81af-12089a542455/cleanup/0.log"
Sep 10 05:18:48.583651 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:48.583614    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78fc98c8bf-kpwwf_a336d026-9d0e-4ac6-81d8-08e992654b98/authorino/0.log"
Sep 10 05:18:49.969575 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:49.969547    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b5848dcdb-f95p8_78923cab-9d82-4bf4-b689-ac5773511687/kube-auth-proxy/0.log"
Sep 10 05:18:50.207905 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:50.207871    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-r72v6_40f884d1-f689-4baa-b91a-53debfb0e56f/payload-pre-processing/0.log"
Sep 10 05:18:50.915799 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:50.915771    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-nvp56_7ecd3422-b133-42ac-9f55-4e486a3d1159/main/0.log"
Sep 10 05:18:51.252216 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:51.252189    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmrr88_63241770-1f42-4b25-a827-b07d4a69f72b/main/0.log"
Sep 10 05:18:51.364181 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:51.364152    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-fjdxn_3fd2862e-d95a-42f5-9924-bbcbf30a31c6/main/0.log"
Sep 10 05:18:51.481137 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:51.481104    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-fgml9_dc12d19c-9c81-4c4f-8ddb-06cf04abfcc1/main/0.log"
Sep 10 05:18:56.522589 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:56.522554    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:18:56.572093 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:56.572017    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:18:57.010257 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:57.010222    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:18:57.111576 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:18:57.111538    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:04.413565 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:04.413532    2575 scope.go:117] "RemoveContainer" containerID="63ca5342046520949a3871d2af2be13ba3d4f82eeb05f0daee5ebf71ece9ff56"
Sep 10 05:19:04.775771 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:04.775740    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78fc98c8bf-kpwwf_a336d026-9d0e-4ac6-81d8-08e992654b98/authorino/0.log"
Sep 10 05:19:08.867890 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:08.867846    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs"]
Sep 10 05:19:08.868296 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:08.868218    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="200688a3-0725-4992-9ec2-176b0afa8fc1" containerName="main"
Sep 10 05:19:08.868296 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:08.868231    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="200688a3-0725-4992-9ec2-176b0afa8fc1" containerName="main"
Sep 10 05:19:08.868296 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:08.868290    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="200688a3-0725-4992-9ec2-176b0afa8fc1" containerName="main"
Sep 10 05:19:08.872747 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:08.872727    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs"
Sep 10 05:19:08.874996 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:08.874966    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x7vl4\"/\"default-dockercfg-b6tnh\""
Sep 10 05:19:08.875258 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:08.875237    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x7vl4\"/\"openshift-service-ca.crt\""
Sep 10 05:19:08.876057 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:08.876035    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x7vl4\"/\"kube-root-ca.crt\""
Sep 10 05:19:08.880333 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:08.880307    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs"]
Sep 10 05:19:08.969306 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:08.969266    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97f2249a-a4b4-4523-a03f-9d74ef126050-sys\") pod \"perf-node-gather-daemonset-2z6rs\" (UID: \"97f2249a-a4b4-4523-a03f-9d74ef126050\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs"
Sep 10 05:19:08.969306 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:08.969305    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97f2249a-a4b4-4523-a03f-9d74ef126050-lib-modules\") pod \"perf-node-gather-daemonset-2z6rs\" (UID: \"97f2249a-a4b4-4523-a03f-9d74ef126050\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs"
Sep 10 05:19:08.969537 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:08.969325    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb8kb\" (UniqueName: \"kubernetes.io/projected/97f2249a-a4b4-4523-a03f-9d74ef126050-kube-api-access-gb8kb\") pod \"perf-node-gather-daemonset-2z6rs\" (UID: \"97f2249a-a4b4-4523-a03f-9d74ef126050\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs"
Sep 10 05:19:08.969537 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:08.969386    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/97f2249a-a4b4-4523-a03f-9d74ef126050-proc\") pod \"perf-node-gather-daemonset-2z6rs\" (UID: \"97f2249a-a4b4-4523-a03f-9d74ef126050\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs"
Sep 10 05:19:08.969537 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:08.969454    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/97f2249a-a4b4-4523-a03f-9d74ef126050-podres\") pod \"perf-node-gather-daemonset-2z6rs\" (UID: \"97f2249a-a4b4-4523-a03f-9d74ef126050\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs"
Sep 10 05:19:09.070682 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:09.070644    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97f2249a-a4b4-4523-a03f-9d74ef126050-sys\") pod \"perf-node-gather-daemonset-2z6rs\" (UID: \"97f2249a-a4b4-4523-a03f-9d74ef126050\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs"
Sep 10 05:19:09.070682 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:09.070683    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97f2249a-a4b4-4523-a03f-9d74ef126050-lib-modules\") pod \"perf-node-gather-daemonset-2z6rs\" (UID: \"97f2249a-a4b4-4523-a03f-9d74ef126050\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs"
Sep 10 05:19:09.070893 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:09.070702    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb8kb\" (UniqueName: \"kubernetes.io/projected/97f2249a-a4b4-4523-a03f-9d74ef126050-kube-api-access-gb8kb\") pod \"perf-node-gather-daemonset-2z6rs\" (UID: \"97f2249a-a4b4-4523-a03f-9d74ef126050\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs"
Sep 10 05:19:09.070893 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:09.070780    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97f2249a-a4b4-4523-a03f-9d74ef126050-sys\") pod \"perf-node-gather-daemonset-2z6rs\" (UID: \"97f2249a-a4b4-4523-a03f-9d74ef126050\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs"
Sep 10 05:19:09.070893 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:09.070810    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/97f2249a-a4b4-4523-a03f-9d74ef126050-proc\") pod \"perf-node-gather-daemonset-2z6rs\" (UID: \"97f2249a-a4b4-4523-a03f-9d74ef126050\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs"
Sep 10 05:19:09.070893 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:09.070829    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97f2249a-a4b4-4523-a03f-9d74ef126050-lib-modules\") pod \"perf-node-gather-daemonset-2z6rs\" (UID: \"97f2249a-a4b4-4523-a03f-9d74ef126050\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs"
Sep 10 05:19:09.070893 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:09.070848    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/97f2249a-a4b4-4523-a03f-9d74ef126050-proc\") pod \"perf-node-gather-daemonset-2z6rs\" (UID: \"97f2249a-a4b4-4523-a03f-9d74ef126050\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs"
Sep 10 05:19:09.071054 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:09.070901    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/97f2249a-a4b4-4523-a03f-9d74ef126050-podres\") pod \"perf-node-gather-daemonset-2z6rs\" (UID: \"97f2249a-a4b4-4523-a03f-9d74ef126050\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs"
Sep 10 05:19:09.071054 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:09.071002    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/97f2249a-a4b4-4523-a03f-9d74ef126050-podres\") pod \"perf-node-gather-daemonset-2z6rs\" (UID: \"97f2249a-a4b4-4523-a03f-9d74ef126050\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs"
Sep 10 05:19:09.078214 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:09.078188    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb8kb\" (UniqueName: \"kubernetes.io/projected/97f2249a-a4b4-4523-a03f-9d74ef126050-kube-api-access-gb8kb\") pod \"perf-node-gather-daemonset-2z6rs\" (UID: \"97f2249a-a4b4-4523-a03f-9d74ef126050\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs"
Sep 10 05:19:09.182609 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:09.182523    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs"
Sep 10 05:19:09.450515 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:09.450489    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs"]
Sep 10 05:19:09.452900 ip-10-0-132-70 kubenswrapper[2575]: W0910 05:19:09.452871    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod97f2249a_a4b4_4523_a03f_9d74ef126050.slice/crio-95b2271188093116153a44c4b407842c28d1c13d0032be2aad99ac554203182d WatchSource:0}: Error finding container 95b2271188093116153a44c4b407842c28d1c13d0032be2aad99ac554203182d: Status 404 returned error can't find the container with id 95b2271188093116153a44c4b407842c28d1c13d0032be2aad99ac554203182d
Sep 10 05:19:09.470003 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:09.469971    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs" event={"ID":"97f2249a-a4b4-4523-a03f-9d74ef126050","Type":"ContainerStarted","Data":"95b2271188093116153a44c4b407842c28d1c13d0032be2aad99ac554203182d"}
Sep 10 05:19:10.476740 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:10.476701    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs" event={"ID":"97f2249a-a4b4-4523-a03f-9d74ef126050","Type":"ContainerStarted","Data":"58bdf60887f8339e30ddd6bdbfb4700b9e3b5eb41ddbc3b65331085c01e3138b"}
Sep 10 05:19:10.477159 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:10.476770    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs"
Sep 10 05:19:10.492156 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:10.492108    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs" podStartSLOduration=2.492092563 podStartE2EDuration="2.492092563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 05:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:19:10.490324126 +0000 UTC m=+2287.252093335" watchObservedRunningTime="2026-09-10 05:19:10.492092563 +0000 UTC m=+2287.253861742"
Sep 10 05:19:16.491875 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:16.491846    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-2z6rs"
Sep 10 05:19:17.916647 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:17.916614    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b5848dcdb-f95p8_78923cab-9d82-4bf4-b689-ac5773511687/kube-auth-proxy/0.log"
Sep 10 05:19:17.967413 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:17.967387    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-r72v6_40f884d1-f689-4baa-b91a-53debfb0e56f/payload-pre-processing/0.log"
Sep 10 05:19:19.690749 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:19.690716    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b5848dcdb-f95p8_78923cab-9d82-4bf4-b689-ac5773511687/kube-auth-proxy/0.log"
Sep 10 05:19:19.744982 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:19.744949    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-r72v6_40f884d1-f689-4baa-b91a-53debfb0e56f/payload-pre-processing/0.log"
Sep 10 05:19:42.012707 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:42.012667    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:42.112843 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:42.112778    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:42.181622 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:42.181584    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:42.261706 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:42.261668    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:42.342032 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:42.341939    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:42.450116 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:42.450051    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pclmn_920551bb-5e01-45e3-80cd-87f46761fb29/global-pull-secret-syncer/0.log"
Sep 10 05:19:42.489731 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:42.489697    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log"
Sep 10 05:19:42.593978 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:42.593870    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-70.ec2.internal_9d5f561d6e275e920a4621f246ba503e/haproxy/0.log"
Sep 10 05:19:42.936766 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:42.936680    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bbztc_dc941e6f-c3da-4ed4-b77a-3d3157936f7f/node-exporter/0.log"
Sep 10 05:19:42.952851 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:42.952819    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:42.961831 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:42.961808    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bbztc_dc941e6f-c3da-4ed4-b77a-3d3157936f7f/kube-rbac-proxy/0.log"
Sep 10 05:19:42.981165 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:42.981144    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bbztc_dc941e6f-c3da-4ed4-b77a-3d3157936f7f/init-textfile/0.log"
Sep 10 05:19:43.053844 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:43.053805    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:43.440175 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:43.440133    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:43.533394 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:43.533360    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:44.693569 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:44.693537    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-n2rcz_29a6dafa-c02d-4b50-a299-f6b873560b52/networking-console-plugin/0.log"
Sep 10 05:19:45.597398 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:45.597344    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:47.552191 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:47.552159    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mmwjm_b748ea1b-814f-474d-9991-2668883967a3/dns/0.log"
Sep 10 05:19:47.571498 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:47.571472    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mmwjm_b748ea1b-814f-474d-9991-2668883967a3/kube-rbac-proxy/0.log"
Sep 10 05:19:47.688656 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:47.688628    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2kswj_1a1db6ec-8b9d-4f0a-8bf0-de72e290cdd1/dns-node-resolver/0.log"
Sep 10 05:19:47.980707 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:47.980671    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:48.080315 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:48.080281    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:48.378310 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:48.378233    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7d9cd496f5-6fkl6_ff2cb19d-c6a1-4a02-9b91-f2d3841ac177/registry/0.log"
Sep 10 05:19:48.459107 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:48.459082    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n2zlm_02eb37ae-67ce-4454-b34a-e3666a73f999/node-ca/0.log"
Sep 10 05:19:49.891933 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:49.891899    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b5848dcdb-f95p8_78923cab-9d82-4bf4-b689-ac5773511687/kube-auth-proxy/0.log"
Sep 10 05:19:49.963122 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:49.963089    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-r72v6_40f884d1-f689-4baa-b91a-53debfb0e56f/payload-pre-processing/0.log"
Sep 10 05:19:50.958107 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:50.958048    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h96km_3d5779c5-a38f-4b51-841f-896b6c7d873b/serve-healthcheck-canary/0.log"
Sep 10 05:19:51.195987 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:51.195954    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:51.261552 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:51.261466    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:51.352058 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:51.352021    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:51.437037 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:51.437007    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:51.505960 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:51.505913    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:51.633664 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:51.633566    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:51.743592 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:51.743548    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:51.833857 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:51.833824    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:51.839875 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:51.839852    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-srmv2_6440e031-173d-40dd-8186-b80bb9fb3fdb/kube-rbac-proxy/0.log"
Sep 10 05:19:51.870822 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:51.870792    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-srmv2_6440e031-173d-40dd-8186-b80bb9fb3fdb/exporter/0.log"
Sep 10 05:19:51.903876 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:51.903794    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-srmv2_6440e031-173d-40dd-8186-b80bb9fb3fdb/extractor/0.log"
Sep 10 05:19:52.256599 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:52.256564    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:52.357503 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:52.357467    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:52.556843 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:52.556749    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:52.656280 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:52.656240    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:52.757030 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:52.756989    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:53.453239 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:53.453207    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:53.576683 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:53.576650    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:53.849724 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:53.849691    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:53.949878 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:53.949842    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:54.244529 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:54.244498    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:54.466782 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:54.466741    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c69jd_7fc6a301-6f28-4a4f-b547-2991c1cbb8b6/konnectivity-agent/0.log" limit=1024
Sep 10 05:19:54.673563 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:54.673474    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-6r8nx_2bfb57ba-1de6-45ef-9e88-28d442ad2131/server/0.log"
Sep 10 05:19:54.799343 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:54.799311    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-jxfkk_822d9fab-f302-4685-93e1-ef41d09cfd1a/manager/0.log"
Sep 10 05:19:55.016384 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:55.016353    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c57f4866c-qztwq_8ee33fc9-f1e7-4b32-af9f-d3a724d2760e/manager/0.log"
Sep 10 05:19:55.030398 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:55.030374    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c57f4866c-qztwq_8ee33fc9-f1e7-4b32-af9f-d3a724d2760e/copy-manifests/0.log"
Sep 10 05:19:56.154670 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:19:56.154641    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5fb77b59f6-pkxxf_e92dba81-34ab-42c7-8601-f0a0332145b7/manager/0.log"
Sep 10 05:20:00.630956 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:20:00.630914    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-nwwqh_c16defdf-d7ad-44ac-abfe-8c3d4f2199f9/migrator/0.log"
Sep 10 05:20:00.648950 ip-10-0-132-70 kubenswrapper[2575]: I0910 05:20:00.648924    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-nwwqh_c16defdf-d7ad-44ac-abfe-8c3d4f2199f9/graceful-termination/0.log"